System and techniques for patch color correction for an immersive content production system

ABSTRACT

A computing device in communication with an immersive content generation system can generate and present images of a virtual environment on one or more light-emitting diode (LED) displays at least partially surrounding a performance area. The device may capture a plurality of images of a performer or a physical object in the performance area along with at least some portion of the images of the virtual environment by a taking camera. The device may identify a color mismatch between a portion of the performer or the physical object and a virtual image of the performer or the physical object in the images of the virtual environment. The device may generate a patch for the images of the virtual environment to correct the color mismatch. The device may insert the patch into the images of the virtual environment. Also, the device may generate content based on the plurality of captured images.

RELATED APPLICATION(S)

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 63/179,027, filed Apr. 23, 2021, entitled “Color And LightingAdjustment For Immersive Content Production System,” which is hereinincorporated by reference in its entirety and for all purposes.

This application is related to concurrently filed U.S. Non-provisionalapplication Ser. No. 17/716,333, entitled “Color and Lighting Adjustmentfor Immersive Content Production Systems” by Jutan et al., which isincorporated herein by reference. This application is related toconcurrently filed U.S. Non-provisional application Ser. No. 17/716,384,entitled “Color and Lighting Adjustment for Immersive Content ProductionSystems” by Hirschfield et al., which is incorporated herein byreference. This application is related to concurrently filed U.S.Non-provisional application Ser. No. 17/716,474, entitled “Color andLighting Adjustment for Immersive Content Production Systems” by Jutanet al., which is incorporated herein by reference.

FIELD

The present disclosure generally relates to generating content using oneor more displays configured for operation in an immersive contentproduction system. In some embodiments, the images shown on the displaysmay be color corrected in real-time or at interactive frame rates. Inaddition, virtual lights can be generated using the interactivedisplays.

BACKGROUND

One method of creating a virtual reality experience can includesurrounding a user with large display screens that present a virtualenvironment for the user. For example, an immersive content productionsystem that can be used in production of movies and videos can include astage or performance area that is at least partially enclosed with oneor more walls and/or a ceiling each of which can be covered with displayscreens. One or more cameras can be placed in the performance area andlive actors can interact with physical props placed on the stage, aswell as with virtual elements displayed on the displays. Such immersivecontent production systems can present challenges.

SUMMARY

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions.

In one general aspect, a method of content production may includereceiving a first input from a user indicating a lighting value. Themethod can include receiving a second input indicating a region of animmersive virtual environment to which the lighting value is to beapplied. The method can also include applying the lighting value to theregion of the immersive virtual environment. The method can includeoutputting one or more images of the immersive virtual environment, theone or more images based, in part, on the input lighting value. Otherembodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. Themethod may include receiving a selection of an image for a virtual lightand generating the image for the virtual light in the region of theimmersive virtual environment to which the lighting value is to beapplied. In various embodiments, the lighting value may include anintensity value. In various embodiments, the lighting value may includea color value. In various embodiments, the lighting value may include ashape value. In various embodiments, the lighting value may include asoftness value. In various embodiments, the method may include:receiving a third input from an user indicating a size value, the sizevalue may include a height value, a width value, and a rotation value;applying the size value to the region of the immersive virtualenvironment; and outputting one or more images of the immersive virtualenvironment, the one or more images based, in part, on the size value.

In one general aspect, a non-transitory computer-readable medium mayinclude one or more instructions that, when executed by one or moreprocessors of a computing device, cause the computing device to performoperations that can include receiving a first input from a userindicating a lighting value. The operations can include receiving asecond input indicating a region of an immersive virtual environment towhich the lighting value is to be applied. The operations can includeapplying the lighting value to the region of the immersive virtualenvironment. The operations can include outputting one or more images ofthe immersive virtual environment, the one or more images based, inpart, on the input lighting value. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

Implementations may include one or more of the following features. Thenon-transitory computer-readable medium where the one or moreinstructions further cause the computing device to perform operationsincluding receiving a selection of an image for a virtual light andgenerating the image for the virtual light in the region of theimmersive virtual environment to which the lighting value is to beapplied. In various embodiments, the lighting value may include anintensity value. In various embodiments, the lighting value may includea color value. In various embodiments, the lighting value may include ashape value. In various embodiments, the lighting value may include asoftness value. In various embodiments, the one or more instructionsfurther cause the computing device to perform operations includingreceiving a third input from an user indicating a size value, the sizevalue may include a height value, a width value, and a rotation value;apply the size value to the region of the immersive virtual environment;and output one or more images of the immersive virtual environment, theone or more images based, in part, on the size value.

In one general aspect, a computing device may include one or morememories. The computing device may in addition include one or moreprocessors, communicatively coupled to the one or more memories,configured to perform operations. The operations can include receiving afirst input from a user indicating a lighting value. The operations caninclude receiving a second input indicating a region of an immersivevirtual environment to which the lighting value is to be applied. Theoperations can include receiving applying the lighting value to theregion of the immersive virtual environment. The operations can includeoutputting one or more images of the immersive virtual environment, theone or more images based, in part, on the input lighting value.

In various embodiments, the operations can include receiving a selectionof an image for a virtual light and generating the image for the virtuallight in the region of the immersive virtual environment to which thelighting value is to be applied.

In one aspect, a method may include generating a first set of userinterface elements configured to receive a first selection of a shape ofa virtual stage light. The method may include generating a second set ofuser interface elements configured to receive a second selection of animage for the virtual stage light. The method may also includegenerating a third set of user interface elements configured to receivea third selection of a position and an orientation of the virtual stagelight. The method may further include generating a fourth set of userinterface elements configured to receive a fourth selection of a colorfor the virtual stage light. Other embodiments of this aspect includecorresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

Implementations may include one or more of the following features. Invarious embodiments, the method may include generating a fifth set ofuser interface elements configured to receive a fifth selection of atleast one of an intensity, an exposure, a softness, and a roundness ofthe virtual stage light. In various embodiments, the shape of thevirtual stage light may include one of a circle, a square, or atriangle. In various embodiments, the method may include generating aplurality of software switches. The software switches may include atleast one of an active switch, an order switch, and a wrapping switch.In various embodiments, the plurality of software switches may includeat least one of a show border switch, a color and opacity switch, adisplay transform switch, and a gamut transformation switch. In variousembodiments, the third set of user interface elements configured toreceive the third selection of the position and an orientation of thevirtual stage light may include at least one of a rotation wheel, alatitude slider, a longitude wheel, a height slider, and a width slider.In various embodiments, the fourth set of user interface elements can beconfigured to receive the fourth selection of the color for the virtualstage light may include at least one of a color wheel, a plurality ofRed Green Blue sliders, a plurality of Hue Saturation Value sliders, anda color temperature slider.

In one general aspect, method may include generating a first set of userinterface elements configured to receive a first selection of an offsetvalue for adjusting a color of a virtual object. The method may includegenerating a second set of user interface elements configured to receivea second selection of a gamma value for adjusting the color of thevirtual object. The method may include generating a third set of userinterface elements configured to receive a third selection of a gainvalue for adjusting the color of the virtual object. The method mayfurther include generating a software switch for receiving a selectionto enable or disable the offset value, the gamma value, and the gainvalue adjustments to the color of the virtual object. Other embodimentsof this aspect include corresponding computer systems, apparatus, andcomputer programs recorded on one or more computer storage devices, eachconfigured to perform the actions of the methods.

Implementations may include one or more of the following features. Invarious embodiments, the method may include generating a fourth set ofuser interface elements configured to receive a fourth selection of atleast one of an exposure, a saturation, and a contrast of the virtualobject. In various embodiments, the method may include generating afifth set of user interface elements configured to receive a fifthselection of at least one of a mix, a softness, and a roundness of thevirtual object. In various embodiments the method may include generatinga plurality of software switches. The plurality of software switches mayinclude at least one of a bounds switch, an all projectors switch, andan all cameras switch. In various embodiments, at least one of thefirst, the second, or the third set of user interface elements mayinclude at least one of a color wheel, a plurality of Red Green Bluesliders, a plurality of Hue Saturation Value sliders, and a colortemperature slider. In various embodiments, the method may includegenerating a brightness slider for at least one of the first set of userinterface elements, the second set of user interface elements, and thethird set of user interface elements.

In one aspect, a method may include generating a first set of userinterface elements configured to receive a first selection of a color ofa panosphere. A panosphere can be a 360-degree image that can beprojected onto the surface of the walls or LED displays. In variousembodiments, the method may in addition include generating a second setof user interface elements configured to receive a second selection of aposition and an orientation of the panosphere. In various embodiments,the method may also include generating a window containing preview thepanosphere. Other embodiments of this aspect include correspondingcomputer systems, apparatus, and computer programs recorded on one ormore computer storage devices, each configured to perform the actions ofthe methods.

Implementations may include one or more of the following features. Invarious embodiments, the method may include generating a third set ofuser interface elements configured to receive a third selection of atleast one of an exposure, a saturation, and a contrast of thepanosphere. In various embodiments, the first set of user interfaceelements can be configured to receive the first selection of the colorfor the panosphere may include at least one of a color wheel, aplurality of Red Green Blue sliders, a plurality of Hue Saturation Valuesliders, and a color temperature slider. In various embodiments, thesecond set of user interface elements can be configured to receive thesecond selection of the position and an orientation of the panospheremay include at least one of a pan wheel, a tilt wheel, and a heightslider. In various embodiments, the method may include generating aswitch to receive a selection to invert movement of direction of atleast one of second set of user interface elements for the panosphere.In various embodiments, the method may include generating a switch toreceive a selection for a fade to gray transition effect of thepanosphere. In various embodiments, the method may include generating asoftware switch for receiving a selection to enable or disableadjustments to the panosphere. Implementations of the describedtechniques may include hardware, a method or process, or a computertangible medium.

In one general aspect, a computer-implemented method may includegenerating and presenting images of a virtual environment on one or morelight-emitting diode (LED) displays at least partially surrounding aperformance area. The computer-implemented method may include capturinga plurality of images of a performer or a physical object in theperformance area along with at least some portion of the images of thevirtual environment by a taking camera. The computer-implemented methodmay include identifying a color mismatch between a portion of theperformer or the physical object and a virtual image of the performer orthe physical object in the images of the virtual environment. Thecomputer-implemented method may include generating a patch for theimages of the virtual environment to correct the color mismatch. Thecomputer-implemented method may include inserting the patch into theimages of the virtual environment. The computer-implemented method mayinclude generating content based on the plurality of captured images.Other embodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.

In various embodiments, the patch can be displayed flat with respect tothe one or more LED displays. In various embodiments, thecomputer-implemented method may include adjusting an intensity of thepatch displayed on the one or more LED displays. In various embodiments,the computer-implemented method may include adjusting a position of thepatch displayed on the one or more LED displays. In various embodiments,the computer-implemented method may include adjusting an orientation ofthe patch displayed on the one or more LED displays. In variousembodiments, the computer-implemented method may include adjusting anexposure of the patch displayed on the one or more LED displays. Invarious embodiments, the computer-implemented method may includeadjusting a size of the patch displayed on the one or more LED displays.

In an aspect of the disclosure, a non-transitory computer-readablemedium may include one or more instructions that, when executed by oneor more processors of a computing device, cause the computing device toperform operations. The operations can include generating and presentingimages of a virtual environment on one or more light-emitting diode(LED) displays at least partially surrounding a performance area. Theoperations can include capturing a plurality of images of a performer ora physical object in the performance area along with at least someportion of the images of the virtual environment by a taking camera. Theoperations can include identifying a color mismatch between a portion ofthe performer or the physical object and a virtual image of theperformer or the physical object in the images of the virtualenvironment. The operations can include generating a patch for theimages of the virtual environment to correct the color mismatch. Theoperations can include inserting the patch into the images of thevirtual environment. The operations can include generating content basedon the plurality of captured images. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

In various embodiments, the patch can be displayed flat with respect tothe one or more LED displays. In various embodiments, the operations caninclude adjusting an intensity of the patch displayed on the one or moreLED displays. In various embodiments, the operations can includeadjusting a position of the patch displayed on the one or more LEDdisplays. In various embodiments, the operations can include adjustingan orientation of the patch displayed on the one or more LED displays.In various embodiments, the operations can include adjusting an exposureof the patch displayed on the one or more LED displays. In variousembodiments, the operations can include adjusting a size of the patchdisplayed on the one or more LED displays.

In one aspect, an immersive content generation system may include one ormore memories. Immersive content generation system may in additioninclude one or more processors, communicatively coupled to the one ormore memories, configured to perform operation. The operations caninclude generating and presenting images of a virtual environment on oneor more light-emitting diode (LED) displays at least partiallysurrounding a performance area. The operations can include capturing aplurality of images of a performer or a physical object in theperformance area along with at least some portion of the images of thevirtual environment by a taking camera. The operations can includeidentifying a color mismatch between a portion of the performer or thephysical object and a virtual image of the performer or the physicalobject in the images of the virtual environment. The operations caninclude generating a patch for the images of the virtual environment tocorrect the color mismatch. The operations can include inserting thepatch into the images of the virtual environment. The operations caninclude generating content based on the plurality of captured images.Other embodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.

In various embodiments, the patch is displayed flat with respect to theone or more LED displays. In various embodiments, the operations caninclude adjusting an intensity of the patch displayed on the one or moreLED displays. In various embodiments, the operations can includeadjusting a position of the patch displayed on the one or more LEDdisplays. In various embodiments, the operations can include adjustingan orientation of the patch displayed on the one or more LED displays.In various embodiments, the operations can adjusting an exposure of thepatch displayed on the one or more LED displays.

In one general aspect, an apparatus may include a housing enclosing acircuitry. The circuitry may include a processor and a memory. Thehousing can form a handgrip. The apparatus can include a plurality oflight sensors arranged in a particular configuration. Each of theplurality of light sensors can be coupled to an exterior the housing viaa sensor arm. The apparatus can include one or more controls mounted onthe exterior of the housing. The controls can be electrically coupled tothe circuitry. Other embodiments of this aspect include correspondingcomputer systems, apparatus, and computer programs recorded on one ormore computer storage devices, each configured to perform the actions ofthe methods.

Implementations may include one or more of the following features. Theapparatus can include one or more antenna mounted on an exterior of thehousing. The apparatus can include a transmitter connected to thecircuitry and electrically connected to the one or more antenna to senddata from the apparatus via a wireless protocol. In various embodiments,the wireless protocol can be one of Bluetooth, Bluetooth Low Energy, orWi-Fi. In various embodiments, the apparatus can include an electronicdevice for mounting an electronic device to the housing. The electronicdevice can be configured to execute an application for an immersivecontent generation system. In various embodiments, the apparatus mayinclude one or more universal mounts coupled to the exterior of thehousing. In various embodiments, the particular configuration includes afirst light sensor mounted to a first sensor arm that is mountedperpendicular to the handgrip on an opposite side of the housing. Theparticular configuration can include a plurality of additional sensorarms with additional light sensors mounted thereon. The additionalsensor arms can be mounted to the opposite side of the housing of thehandgrip. The additional sensor arms can be mounted at distinct anglesabove and below the first sensor arm. The additional light sensors canbe distributed in a distinct pattern. In various embodiments, theapparatus may include a trigger switch formed as part of the handgrip.The trigger switch can be coupled with the circuitry. In variousembodiments, the plurality of light sensors can be active sensors.

In one aspect, a light capture device can include a body enclosing acircuitry. The circuitry can include a processor and a memory. The bodycan form a handgrip. In various embodiments, the light capture devicecan include a plurality of light sensors arranged in a particularconfiguration. Each of the plurality of light sensors can be coupled toan exterior the body via a sensor arm. The light capture device caninclude one or more controls mounted on the exterior of the body andelectrically coupled to the circuitry. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

In various embodiments, the light capture device can include one or moreantenna mounted on an exterior of the body. In various embodiments, thelight capture device can include a transmitter connected to thecircuitry and electrically connected to the one or more antenna to senddata from the light capture device via a wireless protocol. In variousembodiments, the wireless protocol can be one of Bluetooth, BluetoothLow Energy, or Wi-Fi. In various embodiments, the light capture devicemay include an electronic device mount configured to attach anelectronic device to the body. The electronic device can be configuredto execute an application for an immersive content generation system. Invarious embodiments the light capture device may include one or moreuniversal mounts coupled to the exterior of the body. The universalmounts can be configured for mounting a crown of light sensors. Thecrown can include a plurality of light sensors. In various embodiments,the particular configuration includes a first light sensor mounted to afirst sensor arm that is mounted perpendicular to the handgrip on anopposite side of the body. The particular configuration can include aplurality of additional sensor arms with additional light sensorsmounted thereon. The additional sensor arms can be mounted to theopposite side of the body of the handgrip. The additional sensor armscan be mounted at distinct angles above and below the first sensor arm.The additional light sensors can be distributed in a distinct pattern.In various embodiments, the light capture device can include a triggerswitch formed as part of the handgrip. The trigger switch can be coupledwith the circuitry. In various embodiments, the plurality of lightsensors can be active sensors. Implementations of the describedtechniques may include hardware, a method or process, or a computertangible medium.

In one general aspect, a light capture device may include a housingmolded to form a handgrip enclosing a circuit. The circuit may includeone or more processors connected to a bus, a memory connected to thebus, and a wireless transceiver with at least one antenna connected tothe bus. The light capture device may also include a battery. The lightcapture device may include a plurality of markers arranged in apredetermined configuration. Each of the plurality of markers can becoupled to an exterior surface the housing via a sensor arm. The lightcapture device can include one or more controls mounted on the exteriorsurface of the housing and electrically coupled to the circuit. Otherembodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.

In various embodiments, the wireless transceiver can transmit data via awireless protocol that can include at least one of Bluetooth, BluetoothLow Energy, or Wi-Fi. The light capture device can include an electronicdevice mount configures to removably couple an electronic device to theexterior surface of the housing. The electronic device can be configuredto execute an application for an immersive content generation system.

In various embodiments, the light capture device can include one or moreuniversal mounts coupled to the exterior surface of the housing. Theuniversal mounts can be configured for mounting an accessory. In variousembodiments, the predetermined configuration can include a first lightmarker mounted to a first sensor arm that is mounted perpendicular tothe handgrip on an opposite side of the housing. The predeterminedconfiguration can include a plurality of additional markers arms withadditional markers mounted thereon. The additional sensor arms can bemounted to the opposite side of the housing of the handgrip. Theadditional markers can be mounted at distinct angles above and below thefirst sensor arm. The additional markers can be distributed in adistinct pattern. In various embodiments, the light capture device caninclude a trigger switch formed as part of a handgrip formed as part ofthe exterior surface of the housing. The trigger switch can be coupledwith the circuit. In various embodiments, the plurality of markers canreflect light and are detected by a plurality of motion cameras todetermine a location of the light capture device.

These and other embodiments are described in detail below. For example,other embodiments are directed to systems, devices, and computerreadable media associated with methods described herein.

To better understand the nature and advantages of the present inventionreference should be made to the following description and theaccompanying figures. It is to be understood, however, that each of thefigures is provided for the purpose of illustration only and is notintended as a definition of the limits of the scope of the presentinvention. Also, as a general rule, and unless it is evident to thecontrary from the description, where elements in different figures useidentical reference numbers, the elements are generally either identicalor at least similar in function or purpose.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an immersive content production system according tosome embodiments.

FIG. 2 illustrates an immersive content production system with aperformer on the stage and scenery depicted on the image displaysaccording to some embodiments.

FIG. 3 illustrates an example of the frustum of a taking camera withinthe immersive content production system shown in FIG. 2 .

FIG. 4 illustrates an example of determining a volume of images withinthe frustum of the taking camera within the immersive content productionsystem.

FIG. 5 illustrates an example of a two-dimensional tiling technique fora volume of an image.

FIG. 6 illustrates an example of a three-dimensional volume techniquefor an object in an image.

FIG. 7 illustrates an example of a virtual stage light for the immersivecontent production system.

FIG. 8 is a simplified illustration of the immersive content productionsystem shown in FIG. 3 depicting the frustum of the taking camera from atop view.

FIG. 9 illustrates a simplified illustration of the immersive contentproduction system depicting a virtual stage light.

FIG. 10 is a flowchart of an example process associated with a lightingadjustment for immersive content production system.

FIG. 11 is a flowchart of an example process associated with generationof a graphical user interface for color and lighting adjustment forimmersive content production system.

FIG. 12 is a flowchart of an example process associated with generationof a graphical user interface for color and lighting adjustment forimmersive content production system.

FIG. 13 is a flowchart of an example process associated with generationof a graphical user interface for color and lighting adjustment forimmersive content production system.

FIG. 14 is a flowchart of an example process associated with coloradjustments for a panosphere for an immersive content production system.

FIG. 15 illustrates a trackpad user interface according to variousaspects of the disclosure.

FIG. 16 illustrates a stage positioning panel according to variousaspects of the disclosure.

FIG. 17 illustrates an animation triggers panel according to variousaspects of the disclosure.

FIG. 18 illustrates a virtual stage lights panel according to variousaspects of the disclosure.

FIG. 19 illustrates a virtual stage light/stickers panel according tovarious aspects of the disclosure.

FIG. 20 illustrates a panosphere control panel according to variousaspects of the disclosure.

FIG. 21 illustrates an alternate panosphere control panel according tovarious aspects of the disclosure.

FIG. 22 illustrates a detached virtual green screen panel according tovarious aspects of the disclosure.

FIG. 23 illustrates an attached virtual green screen panel according tovarious aspects of the disclosure.

FIG. 24 illustrates a color correction panel according to variousaspects of the disclosure.

FIG. 25 illustrates a first exemplary handles panel according to variousaspects of the disclosure.

FIG. 26 illustrates a second exemplary handles panel according tovarious aspects of the disclosure.

FIG. 27 illustrates a profile view of a light capture device accordingto various aspects of the disclosure.

FIG. 28 illustrates a top view of the light capture device according tovarious aspects of the disclosure.

FIG. 29 illustrates a front view of the light capture device accordingto various aspects of the disclosure.

FIG. 30 illustrates a profile view of the light capture device accordingto various aspects of the disclosure.

FIG. 31 illustrates a simplified block diagram of an immersive contentproduction system according to some embodiments.

Like reference, symbols in the various drawings indicate like elements,in accordance with certain example implementations. In addition,multiple instances of an element may be indicated by following a firstnumber for the element with a letter or a hyphen and a second number.For example, multiple instances of an element 110 may be indicated as110-1, 110-2, 110-3 etc., or as 110 a, 110 b, 110 c, etc. When referringto such an element using only the first number, any instance of theelement is to be understood (e.g., element 110 in the previous examplewould refer to elements 110-1, 110-2, and 110-3 or to elements 110 a,110 b, and 110 c).

DETAILED DESCRIPTION

The following detailed description of example implementations refers tothe accompanying drawings. The same reference numbers in differentdrawings may identify the same or similar elements.

Embodiments are directed at an immersive content production system thatincludes a plurality of displays. Immersive content (e.g., virtualreality content, mixed reality content, augmented reality content,content configured for immersive caves/walls, etc.) may be leveraged aspart of a system used by users (e.g., artists, engineers, technicians,directors, and other individuals involved in content production) inorder to generate content (e.g., movies, television programming, onlineor streamed videos, etc.). As described herein, the immersive contentproduction system may also be referred to as simply the contentproduction system or production system.

In one aspect, the immersive content production system presents imagesin real-time or at interactive frame rates (e.g., 24, 30, 60, 120, or240 frames per second) to users of the content production system. Theimages may be presented over immersive devices (e.g., virtual realitygoggles and augmented reality glasses) or via an immersive environment,such as an immersive “cave” or one or more immersive “walls” (e.g., aperformance area partially or completely surround with image displays).In one embodiment, the immersive environment may include a performancearea, such as a stage. The performance area may be partially orcompletely surrounded by light emitting diode (LED) or liquid crystaldisplay (LCD) display screens. For example, the performance area mayinclude one or more walls and a ceiling of LED display screens enclosingor surrounding the performance area. Alternatively, the performance areamay be partially or completely surrounded by projector screens. A set ofprojectors may additionally be configured to generate images on theprojector screens. In some embodiments, the performance area may bepartially or completely surrounded by a combination of LED or LCDdisplay screens and projector screens. In some embodiments, the contentproduction system may obtain virtual environment content and display thecontent on the image displays around the performance area. In this way,a performer/actor in the performance area may appear to be within thevirtual environment. In some embodiments, the images displayed by theimages displays are primarily background content (e.g., trees,buildings, the sun, etc.).

In various embodiments, the content production system can include one ormore cameras usable for capturing a performance being performed by aperformer in the performance area. The performance area may be, forexample, a movie/television set, stage, stadium, park, etc. During theperformance, the content production system may detect the motion and/orpositioning of the performer. Such detection may be based on markers orsensors worn by the performer, depth and/or other motion detectionsensors of the content production system (e.g., light detection andranging (LIDAR)), motion capture cameras, etc. For example, an array ofdepth sensors may be positioned in proximity to and directed at theperformance area. For instance, the depth sensors may surround theperimeter of the performance area. In some embodiments, the depthsensors measure the depth of different parts of the performer in theperformance area over the duration of a performance. The depthinformation may then be stored and used by the content production systemto determine the positioning of the performer over the performance.

In certain embodiments, a taking camera can be aimed at the performancearea may capture the performance of the performer as well as the virtualenvironment displayed by the image displays (e.g., LED displays) behindthe performer. In some embodiments, sensors may be used to determine theposition and orientation of the taking camera during a performance. Forexample, Global Navigation Satellite System (GNSS) based sensors may beattached to the taking camera to determine its position within orrelative to the performance area. As another example, other cameras maybe directed at the taking camera configured to capture the performance.One or more markers may be attached to the taking camera. During aperformance, the other cameras may capture images of the taking cameraas the taking camera is moved and/or oriented during the performance.The production system may use the images captured of the taking camerato determine the movement and orientation of the taking camera duringthe performance. Such information may be used to support the contentproduction process. For example, such information regarding theorientation and movement of the taking camera may be used to determinethe distance of the taking camera from the performer over a performance.Based on the orientation and movement (and other attributes such as lensaperture and focal length) of the taking camera, the content productionsystem may adjust the virtual environment displayed by the immersivecave or walls in real-time or at interactive frame rates to correspondto orientation and position of the camera. In this way, images of thevirtual environment can be perspective-correct over a performance of theperformer.

According to some embodiments, methods, and systems of color calibrationand adjustment for a content production system are provided. Forexample, immersive content (e.g., virtual reality content, mixed realitycontent, augmented reality content, and the like) can be leveraged aspart of a content production system used by users (e.g., artists,engineers, technicians, directors, and other individuals involved incontent production) in order to generate content (e.g., movies,television programming, online or streaming videos, and the like). Toconfigure such a content production system, the lighting and colorprovided by displays surrounding the performance area can beadvantageously calibrated and adjusted so that the performers, physicalitems in the performance area, and virtual items shown on the screen canhave a selected lighting and color.

Embodiments described herein provide for applying lighting and or coloradjustments made to the virtual environment. Various graphical userinterfaces are described that can be used by an operator to makeadjustments to color and lighting of virtual environment images,generate virtual stage lights, create attached and unattached greenscreens, generate two-dimensional (2D) virtual objects (e.g., stickers)and/or trigger virtual effects from a tablet computing device. A lightmeasuring device is also disclosed to determine lighting and colorvalues generated by the displays of the immersive cave/wall. The lightmeasuring device may provide information regarding the measured lightand color values to the content production system.

FIG. 1 is a simplified illustration of an immersive content productionsystem 100 according to some embodiments. The immersive contentproduction system 100 can include a performance area 102 that ispartially or completely surrounded by image displays 104 (also referredto herein as just “displays”). The immersive content production system100 can obtain virtual content and display the virtual content on thedisplays 104.

The performance area 102 can be, for example, a movie or television set,a stage, a stadium, a park, or the like. In one aspect, the immersivecontent production system 100 presents images in real-time or atinteractive frame rates to users of the immersive content productionsystem 100 (e.g., performers within the performance area 102). Since thedisplays 104 surround or partially surround the performance area 102,the immersive content production system 100 can create an immersiveenvironment (also referred to as an immersive “cave” or immersive“walls”) for performances that take place within the performance area102. In this way, an actor or actress performing within the performancearea 102 can appear to be in a virtual environment.

In some embodiments, the displays 104 can include light emitting diode(LED) display screens or liquid crystal display (LCD) display screens.For example, the performance area 102 can include one or more walls ofLED or LCD displays 104 enclosing the performance area 102.Alternatively, the performance area 102 can be partially or completelysurrounded by projector screens. A set of projectors can be configuredto project images on the projector screens. In some embodiments, theperformance area 102 can be surrounded by a combination of LED displayscreens, LCD display screens, and/or projector screens.

According to various embodiments, the displays 104 can have varioussizes, and the performance area 102 can also have various sizes. In someembodiments the displays 104 can be 20-40 feet tall, and the performancearea 102 can be, for example, between 50-100 feet in diameter. In someembodiments, the displays 104 can include multiple displays 104 that aregenerally fixed in position and mostly surround the performance area102, along with additional moveable or mobile displays 104 that can bemoved into positions that create an immersive environment that extendscompletely or almost completely (i.e., 300-360 degrees) around theperformance area 102. As an example, in one embodiment, fixed positiondisplays 104 can extend approximately 270 degrees around the performancearea 102, while moveable displays 104 can be used to augment the fixedposition displays to further extend the immersive environment up to 320degrees or up to 360 degrees around the performance area. Additionally,while not shown in FIG. 1 , in some embodiments, the immersive contentproduction system 100 can further include one or more displays 104 as aceiling on the performance area 102 and/or as part of the floor of theperformance area 102. Also, while for ease of illustration, the displays104 are shown in FIG. 1 as having a small space or gap 105 betweenadjacent displays 104, the displays 104 can be installed so as to beseamless, with less than a threshold distance or even no space betweenadjacent displays 104. In some instances, the displays 104 may becurved.

A taking camera 112 can be attached to a rig 110 and can be aimed at theperformance area 102 to capture the performance of a performer as wellas the virtual environment displayed on the displays 104. In someembodiments, sensors can be used to determine the position andorientation of the taking camera 112 during a performance. For example,GPS based sensors (not shown) can be attached to the taking camera 112to determine its position within or relative to the performance area102.

In some embodiments, other cameras (e.g., motion capture, and/oralignment cameras 122 discussed below) can be directed at the takingcamera 112 and/or configured to capture the performance. One or moremarkers can be attached to the taking camera 112. During a performance,the other cameras can capture images of the taking camera 112 as thetaking camera 112 is moved and oriented during the performance. Theimmersive content production system 100 can use the captured images ofthe taking camera 112 to determine the movement and orientation of thetaking camera 112 during the performance. Such information can be usedto support the content production process. For example, such informationregarding the orientation and movement of the taking camera 112 can beused to determine the distance of the taking camera 112 from a performerover a performance. Based on the orientation and the movement (as wellas other intrinsic attributes such as lens aperture and focal length) ofthe taking camera 112, the immersive content production system 100 canadjust the virtual environment displayed by the immersive cave or wallsin real-time or at interactive frame rates to correspond to theorientation and the position of the taking camera 112. In this way,images of the virtual environment can be made perspective-correct withrespect to the performance of the performer.

In some embodiments, the immersive cave or walls can include one or morelighting elements to provide lighting for performance area 102. Forexample, the immersive cave or walls can include supplemental LED lights106 separate from the displays 104 that can light the performance area102 (including the performer) and create various desired lightingeffects. Thus, the LED lights 106 can have the ability to projectlighting levels of different intensities and project such light fromdifferent locations around the stage. In some embodiments the additionalLED lights 106 can be controlled during a performance in order to changethe intensity of the lighting of performance area 102 (including theperformer).

In some embodiments, additional lighting elements can be created withinone or more portions of the various displays 104 that create the virtualenvironment. For example, instead of depicting the virtual environmentin a portion of one or more of the displays 104 surrounding theperformance area, that portion of the display 104 can simulate an LEDlight 108 that illuminates the performance area 102. The immersivecontent production system 100 can include multiple simulated lights 108.The location of each simulated light 108 on the displays 104 can beselected in order to achieve a desired lighting effect. The selectionand placement of simulated lights 108 can be made by a director,lighting technician or other user of the immersive content productionsystem 100, prior to the performance taking place within the performancearea 102 and being filmed by the taking camera 112. The number and thelocation of the simulated lights 108 can be readily adjusted at any timeduring the performance.

Since the simulated lights 108 are created by the displays 104 and arethus part of the displays 104, such simulated lights 108 are alsoreferred to as “embedded lights” or “virtual lights.” The simulatedlights 108 can be in addition to or instead of the supplemental lights106. In some embodiments, the immersive content production system 100can include simulated lights 108 without any supplemental lights 106. Insome embodiments, the taking camera 112 that is capturing images of theperformance area and/or the camera rigs 110 do not include any attachedlights. For example, in some embodiments the taking camera 112 does notinclude a ring of LED lights or other form of lighting for illuminatingthe performance area 102.

In some embodiments, the immersive content production system 100 canfurther include one or more depth sensors 120 and/or one or morealignment cameras 122. The alignment cameras 122, also referred to asmotion cameras, can capture motions in the performance area 102. Duringa performance, the immersive content production system 100 can detectthe motion and/or the positions and the orientations of the performerswithin the performance area 102. The detection can be based on markersor sensors worn by a performer, as well as by the depth sensors 120and/or by the alignment cameras 122. For example, an array of depthsensors 120 can be positioned in proximity to and directed at theperformance area 102. For instance, the depth sensors 120 can surroundthe perimeter of the performance area. In some embodiments, the depthsensors 120 can measure the depth of different parts of a performer inthe performance area 102 over the duration of a performance. The depthinformation can then be stored and used by the immersive contentproduction system 100 to determine the positions of the performer overthe course of the performance.

The depth sensors 120 can include a motion-sensing input device. Thedepth sensor 120 can include a monochrome complementary metal-oxidesemiconductor (CMOS) sensor and an infrared projector. The infraredprojector can project infrared light throughout the first performancearea 102, and the CMOS sensor can measure the distance of each point ofreflected infrared (IR) radiation in the performance area 102 bymeasuring a time it takes for the emitted infrared light to return tothe CMOS sensor. Software in the depth sensors 120 can process the IRinformation received from the depth sensor 120 and use an artificialintelligence machine-learning algorithm to map the visual data andcreate three-dimensional (3-D) depth models of solid objects in theperformance area 102. For example, the one or more depth sensors 120 canreceive emitted infrared radiation to generate 3-D depth models of aperformer, along with the floor, walls, and/or ceiling of theperformance area 102. In one test embodiment, the performance area 102was surrounded by six to eight Kinect® cameras to capture depthinformation of objects and performers in the performance area 102.

The alignment cameras 122 can be part of a motion capture system thatcan track the movement of performers or objects within the immersivecontent production system 100. The alignment cameras 122 can be used tosupport alignment of virtual assets and physical assets, as described inmore detail below. In some instances, the alignment cameras 122 can beused to track the movement of the taking camera 112 and provide alocation of the taking camera to the immersive content production system100. The immersive content production system 100 can use thisinformation to determine what portion of the displays 104 is to berendered from the tracked position and the perspective of the takingcamera 112.

FIG. 2 is a simplified illustration of an immersive content productionsystem 200 according to some embodiments. The immersive contentproduction system 200 can be similar to the immersive content productionsystem 100, and thus includes many or all of the same components asdescribed with respect to FIG. 1 . As shown in FIG. 2 , the immersivecontent production system 200 can include the performance area 102, thedisplays 104, the simulated lights 108, and the taking camera 112attached to the rig 110.

In FIG. 2 , a performer 210 is also shown within the performance area102. The performance area 102 can include one or more physical props 212(e.g., the snowmobile depicted in FIG. 2 ). Scenery images 214 of avirtual environment can be presented on the displays 104 to generate theimmersive environment in which the performer 210 can conduct his or herperformance (e.g., act out a scene in a movie being produced). In someembodiments, the scenery images 214 can be seamlessly presented acrossseveral displays 104 as described with respect to FIG. 1 . The sceneryimages 214 can include one or more virtual light sources 206 that canbe, for example, an image of a sun, a moon, stars, street lights, orother natural or manmade light sources displayed in the scenery images214.

The scenery images 214 can also provide a background for the videocontent captured by the taking camera 112 (e.g., a visible lightcamera). The taking camera 112 can capture a view of performance area202 from a certain perspective. In some embodiments, the taking camera112 can be stationary, while in some other embodiments, the takingcamera 112 can be mounted to a track 110 that can move the taking camera112 during a performance.

Embodiments of the invention can generate and displayperspective-correct images (as rendered from the tracked position andperspective of taking camera 112) onto portions of the surrounding imagedisplay walls that are within the field of view (i.e., the frustum) ofthe taking camera. Areas of the displays 104 outside the field of viewof taking camera 112 can be displayed according to a global viewperspective. Further details associated with generating and displayingcontent on displays 104 according to two different perspectives inaccordance with some embodiments of the invention are discussed below.

I. Color Correction and Lighting Adjustment

FIGS. 3 and 4 are simplified drawings of immersive environmentproduction system 200 from FIG. 2 . Shown in each of FIG. 3 and FIG. 4is a frustum 318 of taking camera 112 within the content productionsystem 200 that includes displays 104 that at least partially encircle aperformance area 102 with a performer 210. Scenery images 214 can appearon the one or more displays 104. For three-dimensional (3D) graphics,the frustum of a camera, also known as a viewing frustum, can be theregion of space in the modeled world that would appear on video takenfrom the taking camera 112. Thus, the frustum 318 is the field of viewof the taking camera 112. The exact shape of viewing frustum 318 canvary and will depend on the lens of taking camera 112 but typically itis a frustum of a rectangular pyramid.

In creating the immersive environment presented on displays 104,immersive content production system 200 can render the portion withinthe frustum of the taking camera 112 differently than it renders theportion outside the frustum of the taking camera 112. For example,embodiments of the disclosure can render the portion 326 of the displays104 that corresponds to frustum 318 as perspective-correct images thatcan update based on movement of the taking camera 112. For example,taking camera 112 can move during a performance as performer 210 movesor to capture the performer from a different angle. As the taking camera112 moves, portions of the scenery images 214 within the viewing frustum318 can be updated in accordance with the perspective of the camera.Portion 328 of the displays 104 outside of the frustum 318 can berendered from a global view perspective and thus display relativelystatic images that do not change based on the movement of the takingcamera 112.

In some embodiments, the images inside the frustum of the taking camera112 can be at a higher resolution than the images outside the frustum.In some embodiments, the images displayed outside the frustum of thetaking camera 112 can be relatively basic scenery images (e.g., bluesky, green grass, gray sea, or brown dirt.) In some instances thescenery images can be completely static. In other instances the sceneryimages 214 can dynamically change over time providing a more realisticbackground for the performance in the immersive content productionsystem 200. For example, clouds can move slowly across the displays 104,branches of trees can blow in the wind, etc. to create realistic,life-like effects. Further, the scenery images 214 can dynamicallychange over time to represent changes in the environment over time.

In some instances, the colors of virtual objects (e.g., the performer210 or snow cat 310) on a display 104 of the immersive cave or walls maynot match or may become unsynchronized with the real world colors of theactors and physical objects within the performance area 102. Forexample, a performer 210 can be wearing a jacket that appears to be onecolor in the taking camera 112 and appears to be a different color orshade in the virtual environment. This could be a result of changinglighting patterns, light from the displays illuminating the actors, etc.As a result, content generated from images captured by the taking camera112 may not appear realistic to viewers due to color discrepanciesbetween the virtual and physical objects.

A set extension can provide an example of the color distortion for animmersive content generation system. For example, a set may have acheckerboard floor that extends into the display that portrays a biggerfloor than the physical set. At some point the floor meets up againstthe LED wall. The LEDs on the wall will light up the real checkerboardfloor. The lighting can result in a color mismatch unless the immersivecontent system is able to color correct the images in the display 104 tosmooth out the seam between the physical floor and the virtual floorscenery images.

In order to have the colors of the displayed virtual objects, actors,and physical objects match visually, the content production system 200may adjust the color of the virtual objects shown on the display inreal-time (or at interactive frame rates) based on the color of theactors and physical objects in the earlier images (i.e., frames) of agiven performance. In various embodiments, the technical supervisors anddirectors can view the images from the taking camera 112 via one or moremonitors and detect the color mismatch.

More specifically, the content production system 200 may automaticallyor periodically receive input from a user to generate a set ofthree-dimensional (3D) volumes, such as volume 430 shown in FIG. 4 . The3D volumes may surround one or more virtual objects (e.g., a virtualspacecraft 432 in a virtual environment to be displayed on the displays104 of the content production system 200. In various embodiments, allvirtual objects within the volumes can receive color correction asdefined by the parameters of the volume. For example, the immersivecontent production system can take the exposure of the color in adefined volume 430 down by half a stop. In some embodiments, the contentgeneration system 200 may present an input tool to enable the user todraw or otherwise define the boundaries of the volumes 430. In certainembodiments, the content generation system 200 may additionally oralternatively generate volumes 430 automatically. In some instances, theautomatic generation of the volumes 430 may be based on the virtualposition of the virtual objects within the virtual environment, tags ortypes associated with the virtual objects (e.g., all virtual objectswith a certain tag may be contained within the same volume 430), anoverall color or some other attribute of the virtual objects, etc. Thetechnique of identifying the specific volumes that need correction savescomputational power because the system does not need to determine whatareas of the screen may have over lapping pixels and requiring thescreen to be rendered multiple times to apply a color correction.

In an example, if a performer is standing in the middle of theperformance area surrounded by displays showing a virtual world withGreek columns around the performer (e.g., 12 columns surrounding theperformer). Each one of the columns can have a color correction capsulevolume fully enclosing just that one column. The immersive contentsystem could be running 12 different color corrections operations. Usinga brute force method, the system could render the whole screen includingthe entire circle of LEDs 12 times over to account for all 12 of thosecolor correction volumes. Using the techniques disclosed herein, thedisplay area can be segmented up into multiple grid zones with each ofthe columns occupying a number of grids (e.g., three zones wide by tenzones tall). Whenever the system renders one of those grid zones, thesystem need only apply the color corrections for volumes that areoccupied as grid zones. So now the system can apply all 12 of thosecolor corrections in parallel. This process of color correction can bedone in near real-time, with the color correction process beingcompleted in a few milliseconds.

During a performance, the content production system may perform a tiled,deferred color correction pass that handles multiple individual 3D colorcorrection volumes simultaneously, bins the volumes into two-dimensional(2D) tiles in screen space, and then processes the affected pixels ineach tile in order according to a per-volume priority factor. Forexample, FIG. 5 illustrates a volume 430 around a virtual spacecraft432. The volume 430 can be divided into a plurality of tiles 510. Forexample, for an 8K LED wall that is 30 feet wide at 1.22 mm pixel pitchcan have 7680 pixels by 4320 pixels. Using 64 pixel by 64 pixels, thatwould produce 120 by 68 tiles. The tiles 510 are not drawn to scale andare used for demonstrative purpose. In various embodiments, the tilescan be 64 pixels by 64 pixels. The tiles 510 may be used to determineareas in which the color correction needs to be applied. For example,the director may indicate that the color of the ramp for the virtualspacecraft 432 needs to be corrected due to a lighting artifact. Thetiles 510 containing the ramp 512 may be identified by the user and onlythe pixels contained within the identified tiles 512 that include theramp may have the color correction applied, therefore leaving the restof the virtual image of the virtual spacecraft 432 unaffected.

In some embodiments, color correction may be performed by determiningone or more color error values or mismatch values between a physicalreference target (e.g., a virtual spacecraft 432) in the performancearea 102 and a given virtual object. The content production system 200may iteratively or progressively modify color related attributes for thevirtual object until the color error values or mismatch values meet oneor more threshold color correction values. As a simple example, thethreshold color correction value for a color green may be set at a valueof 1. The content production system may continuously color correct thegreen color for a virtual object until the virtual object's mismatchvalue is below 1. The iterative process can be a differential renderwhere the color of the selected tiles are compared to a desired color.When the mismatch is above a predetermined threshold, the color isadjusted on the selected tiles until the comparisons are within thepredetermined threshold. In various embodiments, a user provides asecond input to manual adjust the colors for the selected tiles. Invarious embodiments, the system detects the colors of the selected tilesand compares the detected colors with a color for the physical object.

FIG. 6 illustrates an example of a three-dimensional volume techniquefor an object in an image. FIG. 6 illustrates breaking up the volume 430into multiple, sub-volumes 610 instead of the tiles 510 shown in FIG. 5. Using the same example as described above, the director may indicatethat the color of the ramp for the virtual spacecraft 432 needs to becorrected due to a lighting artifact. In this case, the sub-volumes 610containing the ramp may be identified by the user and only theidentified sub-volumes 610 that include the ramp may have the colorcorrection applied, therefore leaving the rest of the virtual image ofthe virtual spacecraft 432 unaffected.

In various embodiments, there may be insufficient processing power tomake all the color corrections required and thus the corrections can beprioritized. In these cases, the lower priority corrections may not bemade.

In some embodiments, the per-volume priority factor for each volume maybe assigned via user input. In other embodiments, the per-volumepriority factor may be based on one or more criteria or heuristics. Forexample, the content production system may assign higher per-volumepriorities for volumes that take up a larger portion of a given imageframe. As another example, the content production system may assignhigher per-volume priorities to volumes that contain virtual objectsthat are moving at least a threshold velocity or that are moving fasterrelative to other virtual objects. As yet another example, the contentproduction system may assign higher per-volume priorities to volumesthat contain virtual objects of certain shapes. For instance, volumeswith virtual objects containing humans (e.g., background characters) maybe assigned higher per-volume priorities. As yet another example, thecontent protection system may assign higher per-volume priorities tovolumes that are virtually closer to the taking camera. The contentproduction system may determine the closeness of a virtual object basedon the virtual depth of the object. As still another example, thecontent production system may be trained using previously assignedpriority factors for various volume shapes and/or virtual object shapes.Based on the training, the content production system may automaticallyassign per-volume priority factors for volumes in a given performance.

FIG. 7 is a simplified drawing of immersive environment productionsystem 700. FIG. 8 is a simplified top view of production system 700.Immersive environment production system 700 includes many elements thatare the same as or similar to elements described above with respect tosystem 200 described above. The same reference numbers used inconjunction with system 200 are used in in FIGS. 7 and 8 with respect tosystem 700 to indicate like elements and, Thus, for the sake ofconvenience and brevity, details of such like elements are not repeated.Shown in each of FIGS. 7 and 8 is a frustum 318 of taking camera 112within the content production system. As described above forthree-dimensional (3D) graphics, the frustum of a camera, also known asa viewing frustum, can be the region of space in the modeled world thatwould appear on video taken from the camera. Thus, the frustum 318 isthe field of view of the camera 112. The exact shape of viewing frustum318 can vary and will depend on the lens of camera 112 but typically itis a frustum of a rectangular pyramid (hence the name).

If the entirety of scenery 214 is rendered from the tracked position andperspective of the taking camera 112 to present perspective-correctimages across the entirety of surrounding displays 104, in someinstances view-dependent lighting artifacts will be present on thephysical foreground components within the performance area 102 (e.g.,the performers/actors, props, and physical set decorations). As thetaking camera 112 moves, the rendered images on the displays 104 update,which can result in a visual discrepancy between the static physical setin the performance area 102 and the virtual assets of the virtualenvironment rendered by displays 104 on the walls. As a result, lightsources (e.g., virtual sun 206) within the virtual environment mightappear to move across the performers 210, across the props 212, and/oracross various set decorations, just because the taking camera 112 isphysically moving. Thus, it might appear that virtual sun 206 movesrelative to a performer based on movement of the taking camera when infact the position of virtual sun 206 relative to the performer shouldnot change.

To mitigate this visual artifact, two separate renderings can beperformed in some instances by the content production system ofembodiments of the invention when displaying a virtual environment. Amethod of rendering content onto the displays 104 according to someembodiments of the invention, the two renderings can be performedsimultaneously and in real-time. In one rendering, a global view of thevirtual environment (including any virtual assets) is renderedindependent from the perspective of the taking camera. The global viewcan include background or scenery images that create much of the virtualor immersive environment that provides context for the one or moreperformers on stage 102. The global view can also include lightingeffects produced from displays outside the frustum of the taking camera112. Depending on the size of frustum 318, this global view renderingcan be displayed on the majority of the display area within the virtualenvironment.

The system can render the global view from a virtual spherical cameraplaced at a virtual location within the virtual environment based onpredefined criteria. In some embodiments, the placement of the virtualspherical camera can be based on a threshold error value for thelighting of virtual objects within the virtual environment and/orlighting of the physical objects in the performance area. The thresholderror value can indicate a minimal acceptable level of visual inaccuracyof the lighting of the objects within the virtual environment and/orperformance area. In some embodiments, the images of the virtualenvironment rendered by in the global-view rendering can remaincompletely static. In other words, objects within the virtualenvironment might not move or change in location on the displays 104over time or during a performance. In other embodiments, objects withinthe global-view of the virtual environment are not completely static butsimply do not update in response to movement of the taking camera 112.Images of the virtual environment generated during the global-viewrendering can be used for lighting and reflection purposes onto thephysical foreground/performance area.

In another rendering, a portion of the virtual environment is renderedfrom the location and perspective of the taking camera. Theperspective-correct rendering can be completely independent from theglobal-view render and can include performers, props and backgroundscenery within the frustum (e.g., frustum 318) of the taking camera 112.The perspective-correct rendering represents a portion of the virtualenvironment and can thought of as a patch that can be displayed on aportion of displays 104. As the global view can be captured by a virtualspherical camera, discrepancies can exist for images displayed on thedisplays in the background from the spherical camera as compared withimages that captured within the frustum of the taking camera. Therefore,a patch can be created to correct the images in the background displaysthat appear within the frustum of the taking camera. In this way as thetaking camera captures the one or more images with actors, props, andbackground, the background appears to be perspective-correct and do notmove abnormally due to movement of the taking camera.

Embodiments of the invention can combine the patch from theperspective-correct rendering with the global-view render to present thevirtual environment on the content production system without (or withfewer) undesirable lighting effects. For example, images of the virtualenvironment can be generated onto displays 104 and updated over thecourse of a performance so that the perspective of the virtualenvironment displayed compensates for corresponding changes to thepositioning and orientation of the taking camera 112. In someembodiments, rendered content is combined such that the displays 104display the perspective-correct rendering in the portion of the displays104 that is viewable by the frustum 318 of the taking camera 112 and theportions of the displays 104 outside of the frustum 318 of the takingcamera 112 only include the rendered images from the global-view render.

II. Additional Color Correction Techniques

An objective of the color correction techniques is to produce as many incamera finals as possible. This will reduce the cost of post processingvideo products. The techniques can be the equivalent to running a 2D and3D compositing tool (e.g., Nuke) in an interactive environment. Thecolor correction techniques attempt to match the 3D elements in thevirtual environment with the real life set pieces that are physically onthe set. For example, for a science fiction show, an entire spacecraftmay not be built on the set, but just a portion (e.g. a ramp or acockpit) may be physically constructed. The color correction techniqueshelp match the physical portion with the virtual images of the ship thatis presented on the displays to make the video as seamless as possible.The technique can include blending the practical set with digital setsso there is very little to no discrepancies perceptible between the realworld environment and the digital environment on the displays.

There can be a few different types of color correction methods. One typecan be known as a global color correction. For examples where the entirescene is too dark an operator may want to stop (e.g., camera stops)everything up to increase the exposure by some camera stops, to make theentire scene brighter. Global corrections can be made to an entirevirtual works on all the LED walls. For example, the global correctioncan increase the brightness or make it bluer or make it warmer or makeit cooler using color charts.

Color correction can be a very artistic process, and in many cases, theoperator can be an artist with over 20 years of experience making colorcorrection adjustments as part of a post-production process. Aspectsdiscussed herein allow for color correction adjustments to be made inreal time. There can be some cases where the adjustments can beautomated or at least partially automated. For example, an operator maywant to adjust the temperature of a group of virtual streetlamps (e.g.,turn up all the streetlamps from 5500 kelvin temperature to 6500 kelvintemperature). A script can be used for this type of adjustment.

As described above, there can be color correction objects, colorcorrection volumes, and color correction windows, and sky colorcorrection. For color correction objects, the objects in the virtualenvironment can be selected by a pointing tool. For example, there maybe hundreds of streetlamps requiring the same color correction. Thestreetlamp can be designated as a color correction object and the colorcorrection controls can be used to change the color or temperature ofall the streetlamps.

The color correction volumes are similar to a voxel pic. The colorcorrection volume is a literal volume of the cube. The edges of thecolor correction volumes can be rounded. The shape of the colorcorrection volumes can be changed. In various embodiments, the volumescan be spherical a cube, a rectangular prism. The 3-D scene can have itsown 3-D coordinates and can move right to the step in the scene. The 3-Deffects can be complicated because it is projected onto a curved, flat2-D wall. The 3-D virtual scene has volume and characters can virtuallywalk around in the images with some depth.

The color correction volume allows for selection of a volume in that 3-Dspace. For example, it can be an area around where a virtual space shipis parked. If the ship is only a 3-D virtual ship and there is some dirton the ground around the ship that the operator wants to adjust thevisuals, one of these volumes can be used for the ground. In that way,the ground area, and anything within that volume can receive a colorcorrection. Anything outside of the volume would not. In variousembodiments, there are other mechanisms for making different shapes tocombine these squares or cubes together so there can be a sort of ablobby shape that would roughly cover a ship or some other odd shape.The color correction volume does not need to be spherical or a definedprism.

A color correction window is similar to a virtual stage light or otherstage projections. Stage projections can be a 2-D virtual object (e.g.,a sticker) that can be displayed over the images on the LED displays. Asthe taking camera moves around the performance area, the 2-D virtualobject stays pointing straight at the taking camera so it does not bendas the camera moves around in the 3-D scenes. The 2-D virtual objectstays as the normal to the display or stated otherwise stays flat to thewall. Other 3-D objects in the virtual space can have depth so they arenot flat to the wall and can be distorted. By using 2-D virtual objects,the shape of the object does not change based on the position of thecamera.

If the camera is locked off, if the camera is not moving, the operatorcan turn off the low pass system temporarily for the camera. With thecamera being fixed there is no wobble. With the motion capture drivertemporarily switched off, an operator can even pick the camera off astand and move it around and the frustum will not move as it will remainlocked to its previous location. The operator can detect an area thatappears too yellow and the operator may want it to be greener. The colorcorrection window can approach the correction from a 2-D perspective asif the operator was editing a single frame in video. The operator maywant to change out one area or cover over one area and blend it out insome way or another. If the camera does not move, the immersive contentproduction system can maintain this parallax trick of ensuring that itis perspective correct-enough to “sell” the trick and still potentiallyachieve an in-camera final. The color correction window provides thatkind of workflow to these artists. Color correction windows are notlimited for use with locked off camera, but that can be an obvious usecase where the operator can fix something in the scene and make it workfor the very specific camera perspective through the lens of the camera.

In certain embodiments, upon selection of a point of region of theimmersive cave/wall, the content production system may enable a user toselect a color and/or lighting values currently associated with thepoint or region at a given playback time for the virtual scene displayedby the immersive cave/wall. For example, at one minute into the playbackof a virtual scene, a region may be associated with the blue color of avirtual sky. The content production system may select the blue color inresponse to a user selection input. The content production system maythen store the color and/or lighting values for the selected point orregions. Thereafter, the user may select another point or region of theimmersive cave/wall using the mechanisms and/or in the same manner aspreviously described. Upon selection, the content production system maymodify the color and/or lighting values of the selected another point orregion based on the color and/or lighting values of the initiallyselected point or region.

In some embodiments, the modification of the color and/or lightingvalues of the selected another point or region may include blendprocessing. More specifically, the content production system maydetermine the color and/or lighting values of neighboring or nearbyregions surrounding the selected another point or region. Based on thevalues for the neighboring regions and the values for the initiallyselected point or region, the content production system may use ablending processing function such that the color and/or lighting valuesof the selected another point or region may be blended with itssurrounding neighbor regions. In various embodiments, the blendingprocessing function can support linear blending. In various embodiments,the blending processing function can support radial or three dimensionalblending. In this way, the modification of the color and/or lightingvalues may appear more seamless to a viewer.

In some embodiments, a user may directly input a particular color and/orlighting value for a selected region. These include, for example, colortemperature, hue, intensity, exposure, softness, roundness, saturation,etc.

In some embodiments, modification of the point or region may includeselecting one or more individual 3D geometries or meshes of the 3Dvirtual scene displayed by the immersive cave/wall that correspond tothe selected point or region of the immersive cave/wall. In doing so,the modification is made to the virtual objects of the 3D virtual scene.Thereafter, a 2D image is generated based on the modified 3D virtualscene for display on the immersive cave/wall. This is in contrast toother embodiments where the underlying 3D virtual scene is not modified.In those embodiments, a 2D image is first generated from the unmodified3D virtual scene. Thereafter, the 2D image is modified to adjust forlighting/color.

In certain embodiments, selection of the underlying geometry includescasting a virtual ray from a projector camera origin, through theselected point on the immersive cave/wall and intersecting the ray withgeometries existing in the 3D virtual scene displayed by the immersivecave/wall. In other embodiments, the content production system initiallycasts a ray with the wall geometry to place a reticle in the 3D virtualscene. Thereafter, the content production system may ray cast from theprojector camera position through the reticle's 3D position to get thetargeted geometry in the 3D virtual scene. In other words, the systemwould place a reticle on a 3D virtual object in a 3D scene correspondingto the direction and/or orientation of the handheld mechanism.Thereafter, a 2D image would be rendered from the 3D scene including thereticle to be displayed on the immersive cave/wall. This is, incontrast, to the system adding a reticle to a corresponding location ona 2D image displayed on the immersive cave/wall following rendering ofthe 2D image from the unmodified 3D virtual scene. Note that the raycast is always from the projector camera position and not from thehandheld mechanism position. In some embodiments, the content productionsystem is configured to cause the immersive cave/wall to display acamera frustum. One benefit of having the reticle live in 3D space (3Dvirtual scene) exactly on the immersive cave/wall is that the reticleposition in 2D space will not change regardless of what projection isused. As such, the reticle as viewed through a frustum would be inexactly the same position as viewed through the projector camera.

III. Virtual Stage Lighting

The LED displays can be used as a light source for the performance area.In the past, a director would request a stage light be set up in acertain area, with a certain size, and a certain color. This would taketime for the lights to be assembled, a colored gel to be applied, andcooling to be set up for the hot lights. Virtual Stage Lights (alsoknown as light cards) can emulate physical lights but with much greaterflexibility and responsiveness. The virtual stage lights can becontrolled through a graphical user interface (GUI) on a tabletcomputer. For example, the GUI can allow an operator to set up a newlight, adjust the location, adjust the size and shape of the light, andadjust various lighting properties (e.g., brightness, color,temperature, etc.)

In various embodiments, the virtual stage lights can emulate a blackflag which is an area that is fully dark so the background is notcasting light onto the performance area (or actors). In variousembodiments, the virtual stage lights can emulate a net which can be ahalf-light card and a half black flag. This can control the amount oflight on the performance area. The virtual stage lights are not just forillumination but can also be used to control reflections. The virtualstage lights can be used to create interesting and unusual reflectionson a ship or on a character.

Virtual green screens can be a way of using a 2-D object in order tohave a real time way to replace the physical green screen. The greenscreen can be the fabric that flips down on the set and then actors actagainst it, not in the context of a 3-D world because it is just green.The green screen is typically replaced in post-production with visualeffects.

But in some cases, it may not be possible to achieve an in-camera finalfor a shot, but it would be undesirable to just make the whole back wallgreen. Using virtual green screen can provide some of the benefits ofcontrolling the reflections or from the lighting or other projectedscene elements displayed on the LED displays except for maybe onesection. For example, maybe there is an area around an actor someonewith very specific hair. There can be issues with hair and other fineclothing details in general with visual effects compositing. In variousembodiments, it may be desirable to mask out a character, for example ina face replacement. In those cases a green light card or a blue lightcard for a blue screen may also have some digital markers on top of thebase color. Instead of taping up a green screen and putting literal “Xs”or “+” signs made out of tape as on a real “green screen” on a set, avirtual green screen can be generated digitally. The virtual greenscreen can further enhance the ability to get in-camera finals for therest of the shot but not maybe for one section that is potentially beingpinned up.

In the virtual green screen, the operator can adjust lights also. Therecan still be lighting coming off the virtual green screen that may helpwith some of the rest of the image. Lights can be combined with greenscreen so you could either have a green screen on top of or below thelights.

In another case, one stage can have a strong light on one side of thevolume and a very dark region on the wall on the other side so it wasflaring out on the far wall. It was just making it too bright. So whileit was imperfect for the rest of the scene, that particular angleshooting that wall, it was not possible to get an acceptable shot so theoperator chose to use green screen for that.

In other examples, there may be no 3-D elements and the operator mayjust have this wrapped beautiful panosphere view that was captured usinga 360 degree camera in a unique location. That scene is being projectedon the displays and then the operator can layer light cards and colorcorrection or other real-time creative edits on top of that unwrapped360-degree image. That lighting effect can be known as a panosphere. Thepanosphere can receive color, position, and other corrections.

Panosphere is like the sky itself. The sky may be separate from the 3-Dscene. The displays may have a layered sky behind it. For example, thescene may have a desert landscape and then the operator has an image ofsky. The operator may want to change the position of the sun and colorcorrect the sky or rotate the position of the clouds in the skyseparately from the scene.

FIG. 9 illustrates a modification of the immersive content productionsystem of FIG. 2 to include an example of a virtual stage light 902. Theimmersive content production system 900 can be similar to the immersivecontent production system 200, and thus includes many or all of the samecomponents as described with respect to FIG. 2 . As shown in FIG. 9 ,the immersive content production system 900 can include the performancearea 102, the displays 104, the simulated lights 108, and the takingcamera 112 attached to the rig 110. A performer 210 is also shown withinthe performance area 102. The performance area 102 can include one ormore physical props 212 (e.g., the snowmobile depicted in FIG. 2 ).Scenery images 214 of a virtual environment can be presented on thedisplays 104 to generate the immersive environment in which theperformer 210 can conduct his or her performance (e.g., act out a scenein a movie being produced). In some embodiments, the scenery images 214can be seamlessly presented across several displays 104 as describedwith respect to FIG. 1 . The scenery images 214 can include one or morevirtual light sources 206 that can be, for example, an image of a sun, amoon, stars, street lights, or other natural or manmade light sourcesdisplayed in the scenery images 214.

The scenery images 214 can also provide a background for the videocontent captured by the taking camera 112 (e.g., a visible lightcamera). The taking camera 112 can capture a view of performance area202 from a certain perspective. In some embodiments, the taking camera112 can be stationary, while in some other embodiments, the takingcamera 112 can be mounted to a track 110 that can move the taking camera112 during a performance.

Embodiments of the invention can generate and displayperspective-correct images (as rendered from the tracked position andperspective of taking camera 112) onto portions of the surrounding imagedisplay walls that are within the field of view (i.e., the frustum) ofthe taking camera. Areas of the displays 104 outside the field of viewof taking camera 112 can be displayed according to a global viewperspective. Further details of associated with generating anddisplaying content on displays 104 according to two differentperspectives in accordance with some embodiments of the invention arediscussed below.

In some embodiments, the size and/or shape of a selected region may bealtered by the user in real-time or substantially real-time. In someinstances, the light/color adjustment element used for selecting aregion and/or the selected region itself (used for the purpose ofadjusting the lighting of another region) may be referred to as aLightcard. In some embodiments, a gradient may be selected and appliedto a particular region or point. More specifically, two or more colorsand/or lighting values may be selected for a gradient to be applied to aregion. Thereafter, gradient points may be placed on the region thatcorresponds to each selected color and/or light value. In response, thesystem may apply a gradient in which the color and/or lighting valuesare progressively changed from one corresponding color and/or lightingvalue to another corresponding color and/or lighting value between twogradient points.

In some embodiments, the lighting and/or color may be applied based onthe 3D virtual objects corresponding to the region to which the lightingand/or color values is to be applied. Specifically, the system maydetermine/account for the 3D virtual object type in applying the colorand/or lighting. For example, the color and/or lighting may be applieddifferently if the corresponding 3D virtual object type is metal asopposed to grass. In some embodiments, the system may determine aboundary between two 3D virtual objects and blend the two virtualobjects accordingly. For example, the system may determine that aselected region includes a boundary between a virtual sky and virtualground. In response, the system may apply a blending technique to makethe boundary between the two objects more seamless to a viewer.

IV. Virtual Stage Lights Flow

FIG. 10 is a flowchart of an example process 1000 associated with alighting adjustment for immersive content production system. In someimplementations, one or more process blocks of FIG. 10 may be performedby a computing device (e.g., computing device 3100 discussed below withrespect to FIG. 31 ). In some implementations, one or more processblocks of FIG. 10 may be performed by another device or a group ofdevices separate from or including the computing device. Additionally,or alternatively, one or more process blocks of FIG. 10 may be performedby one or more components of computing device 3100, such as processingunit 3104, storage subsystem 3118, system memory 3110, cameras 3134,displays 3132, communications subsystems 3124, and bus 3102.

At block 1010, process 1000 may include receiving a first input from auser indicating a lighting value. For example, the computing device mayreceive a first input from a user indicating a lighting value, asdescribed above. The input can be received via a pointing device or alighting capture device as described later. In various embodiments, thefirst input can be received on a GUI via an application on a tabletcomputer. In various embodiments, the first inputs can be made via alaptop computer or a desktop computer.

In a first implementation, process 1000 includes receiving a selectionof an image for a virtual light, and generating the image for thevirtual light in the region of the immersive virtual environment towhich the lighting value is to be applied. In a second implementation,the lighting value comprises an intensity value. In a thirdimplementation, the lighting value comprises a color value. In a fourthimplementation, the lighting value comprises a shape value. In a fifthimplementation, the lighting value comprises a softness value.

At block 1020, process 1000 may include receiving a second inputindicating a region of an immersive virtual environment to which thelighting value is to be applied. For example, the computing device mayreceive a second input indicating a region of an immersive virtualenvironment to which the lighting value is to be applied, as describedabove. The second input can be received via a pointing device or alighting capture device as described later. In various embodiments, thesecond input can be received on a GUI via an application on a tabletcomputer. In various embodiments, the second input can be made via alaptop computer or a desktop computer.

In various embodiments, process 1000 includes receiving a third inputfrom a user indicating a size value, the size value comprising a heightvalue, a width value, and a rotation value, and applying the size valueto the region of the immersive virtual environment, outputting one ormore images of the immersive virtual environment, the one or more imagesbased, in part, on the size value.

At block 1030, process 1000 may include applying the lighting value tothe region of the immersive virtual environment. For example, thecomputing device may apply the lighting value to the region of theimmersive virtual environment, as described above.

At block 1040, process 1000 may include outputting one or more images ofthe immersive virtual environment, the one or more images based, inpart, on the input lighting value. For example, the computing device mayoutput one or more images of the immersive virtual environment, the oneor more images based, in part, on the input lighting value, as describedabove.

Process 1000 may include additional implementations, such as any singleimplementation or any combination of implementations described belowand/or in connection with one or more other processes describedelsewhere herein.

Although FIG. 10 shows example blocks of process 1000, in someimplementations, process 1000 may include additional blocks, fewerblocks, different blocks, or differently arranged blocks than thosedepicted in FIG. 10 . Additionally, or alternatively, two or more of theblocks of process 1000 may be performed in parallel.

V. Lighting and Color Correction Graphical User Interfaces

In some embodiments, the content production system may enable a user toadjust the color and lighting of the displays 104 in real orsubstantially real-time (e.g., at interactive frame rates). In someembodiments, the content production system may include a mechanism toenable the user to select a region of the displays 104 for which thecolor and/or lighting should be adjusted. In certain embodiments, themechanism may be a tablet or other mobile computing device. FIGS. 15-16provide exemplary graphical user interfaces that can be executed on thetablet or the other mobile computing device.

The mobile computing device may present an interface that includes arepresentation of the immersive cave/wall comprising the displays 104.The representation may be presented as flattened version of theimmersive cave/wall. In some instances, mapping data may be stored andleveraged by the content production system to map the flattenedrepresentation of the immersive cave/wall presented on the mobilecomputing device to the curved/cylindrical physical immersive cave/wall.In some instances, the mapping data may be based on a longitudinal andlatitudinal mapping system. In some instances, the tablet may beconfigured to receive input from a user, such as a finger press orcombination of finger presses. Based on the location of the finger pressreceived at the tablet screen displaying the immersive cave/wallrepresentation, the content production system may select a correspondingregion of or point on the immersive cave/wall.

The GUI may include various layers, windows, screens, templates, orother graphical elements that may be displayed in all, or a portion, ofthe display. Generally, the GUI may include graphical elements thatrepresent applications and functions of the electronic device. Thegraphical elements may include icons and other images representingbuttons, sliders, menu bars, and the like. The icons may correspond tovarious applications of the electronic device that may open uponselection of a respective icon. Furthermore, selection of an icon maylead to a hierarchical navigation process, such that selection of anicon leads to a screen that includes one or more additional icons orother GUI elements. The icons may be selected via a touch screenincluded in the display, or may be selected by a user input structure,such as a wheel or button.

In various embodiments, various input devices can be connected to thetablet computers via either wired or wireless connections (e.g.,Bluetooth). For example, a device having a plurality of physicalsliders, switches (e.g., fader switches), buttons, etc. can be connectedto the tablet computer to provide improved tactile experience.

A touch-sensitive display provides an input interface and an outputinterface between the device and a user. The display controller canreceive and/or sends electrical signals from/to touch screen. The touchscreen displays visual output to the user. The visual output may includegraphics, text, icons, video, and any combination thereof (collectivelytermed “graphics”). In some embodiments, some or all of the visualoutput may correspond to user-interface objects.

The touch screen can include a touch-sensitive surface, sensor, or setof sensors that accepts input from the user based on haptic and/ortactile contact. The touch screen and display controller (along with anyassociated modules and/or sets of instructions stored in memory of thetablet computer) detect contact (and any movement or breaking of thecontact) on the touch screen and convert the detected contact intointeraction with user-interface objects (e.g., one or more soft keys,icons, web pages or images) that are displayed on touch screen. In anexemplary embodiment, a point of contact between the touch screen andthe user corresponds to a finger of the user.

The touch screen may use LCD (liquid crystal display) technology, LED(light emitting polymer display) technology, or LED (light emittingdiode) technology, although other display technologies may be used inother embodiments. The touch screen and the display controller maydetect contact and any movement or breaking thereof using any of aplurality of touch sensing technologies now known or later developed,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies, as well as other proximity sensorarrays or other elements for determining one or more points of contactwith the touch screen. In an exemplary embodiment, projected mutualcapacitance sensing technology can be used.

In some embodiments, in addition to the touch screen, a computing device(e.g., a tablet computer or a notebook computer) may include a touchpad(not shown) for activating or deactivating particular functions. In someembodiments, the touchpad is a touch-sensitive area of the device that,unlike the touch screen, does not display visual output. The touchpadmay be a touch-sensitive surface that is separate from the touch screenor an extension of the touch-sensitive surface formed by the touchscreen.

In some embodiments, the computing device may include a physical orvirtual wheel (e.g., a click wheel) as an input control device. A usermay navigate among and interact with one or more graphical objects(e.g., icons) displayed in the touch screen by rotating the click wheelor by moving a point of contact with the wheel (e.g., where the amountof movement of the point of contact is measured by its angulardisplacement with respect to a center point of the wheel). The wheel mayalso be used to select one or more of the displayed icons. For example,the user may press down on at least a portion of the wheel or anassociated button. User commands and navigation commands provided by theuser via the wheel may be processed by an input controller as well asone or more of the modules and/or sets of instructions in the memory ofthe computing device. For a virtual wheel, the wheel and wheelcontroller may be part of the touch screen and the display controller,respectively. For a virtual wheel, the wheel may be either an opaque orsemitransparent object that appears and disappears on the touch screendisplay in response to user interaction with the device. In someembodiments, a virtual wheel is displayed on the touch screen of aportable multifunction device and operated by user contact with thetouch screen. In various embodiments, haptic feedback can be used on thetablet computer when making changes on the user interface.

A contact/motion module may detect contact with the touch screen (inconjunction with the display controller) and other touch sensitivedevices (e.g., a touchpad or physical click wheel). The contact/motionmodule can include various software components for performing variousoperations related to detection of contact, such as determining ifcontact has occurred (e.g., detecting a finger-down event), determiningif there is movement of the contact and tracking the movement across thetouch-sensitive surface (e.g., detecting one or more finger-draggingevents), and determining if the contact has ceased (e.g., detecting afinger-up event or a break in contact). The contact/motion module canreceive contact data from the touch-sensitive surface. Determiningmovement of the point of contact, which is represented by a series ofcontact data, may include determining speed (magnitude), velocity(magnitude and direction), and/or an acceleration (a change in magnitudeand/or direction) of the point of contact. These operations may beapplied to single contacts (e.g., one finger contacts) or to multiplesimultaneous contacts (e.g., “multi-touch”/multiple finger contacts). Insome embodiments, the contact/motion module and the display controllercan detect contact on a touchpad. In some embodiments, thecontact/motion module and controller can detect contact on a clickwheel.

The contact/motion module may detect a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns. Thus, a gesture may be detected by detecting a particularcontact pattern. For example, detecting a finger tap gesture includesdetecting a finger-down event followed by detecting a finger-up (liftoff) event at the same position (or substantially the same position) asthe finger-down event (e.g., at the position of an icon). As anotherexample, detecting a finger swipe gesture on the touch-sensitive surfaceincludes detecting a finger-down event followed by detecting one or morefinger-dragging events, and subsequently followed by detecting afinger-up (lift off) event.

In various embodiments, augmented reality or virtual reality controllers(e.g., Google Glass, or Oculus controllers) can be used to select andmodify various elements of the virtual environment. Other hand-heldpointers and controllers can be used by the operator.

FIG. 11 is a flowchart of an example process 1100 associated withgeneration of a graphical user interface for color and lightingadjustment for immersive content production system. In someimplementations, one or more process blocks of FIG. 11 may be performedby a computing device (e.g., computing device 3100). Additionally, oralternatively, one or more process blocks of FIG. 10 may be performed byone or more components of device 3100, such as processing unit 3104,storage subsystem 3118, system memory 3110, cameras 3134, displays 3132,communications subsystems 3124, and bus 3102.

At block 1110, process 1100 may include generating a first set of userinterface elements configured to receive a first selection of a shape ofa virtual stage light. For example, the computing device may generate afirst set of user interface elements configured to receive a firstselection of a shape of a virtual stage light, as described above. Invarious implementations, the shape of the virtual stage light comprisesone of a circle, a square, or a triangle. The first selection can bereceived via a GUI on a tablet computing device as described below. Thefirst selection can be received via a light capture device as describedbelow. The first selection can be received via an input/output device ona computing device (e.g., a desktop or laptop computer). The firstselection can be received via a virtual interface using augmentedreality devices or virtual reality devices.

At block 1120, process 1100 may include generating a second set of userinterface elements configured to receive a second selection of an imagefor the virtual stage light. For example, the computing device maygenerate a second set of user interface elements configured to receive asecond selection of an image for the virtual stage light, as describedabove. The second selection can be received via a GUI on a tabletcomputing device as described below. The second selection can bereceived via a light capture device as described below. The secondselection can be received via an input/output device on a computingdevice (e.g., a desktop or laptop computer). The second selection can bereceived via a virtual interface using augmented reality devices orvirtual reality devices.

At block 1130, process 1100 may include generating a third set of userinterface elements configured to receive a third selection of a positionand an orientation of the virtual stage light. For example, thecomputing device may generate a third set of user interface elementsconfigured to receive a third selection of a position and an orientationof the virtual stage light, as described above. The third selection canbe received via a GUI on a tablet computing device as described below.The third selection can be received via a light capture device asdescribed below. The third selection can be received via an input/outputdevice on a computing device (e.g., a desktop or laptop computer). Thethird selection can be received via a virtual interface using augmentedreality devices or virtual reality devices.

In various implementations, the third set of user interface elementsconfigured to receive the third selection of the position and anorientation of the virtual stage light comprise at least one of arotation wheel, a latitude slider, a longitude wheel, a height slider,and a width slider.

At block 1140, process 1100 may include generating a fourth set of userinterface elements configured to receive a fourth selection of a colorfor the virtual stage light. For example, the computing device maygenerate a fourth set of user interface elements configured to receive afourth selection of a color for the virtual stage light, as describedabove. The fourth selection can be received via a GUI on a tabletcomputing device as described below. The fourth selection can bereceived via a light capture device as described below. The fourthselection can be received via an input/output device on a computingdevice (e.g., a desktop or laptop computer). The fourth selection can bereceived via a virtual interface using augmented reality devices orvirtual reality devices.

In various implementations, the fourth set of user interface elementsconfigured to receive the fourth selection of the color for the virtualstage light comprise at least one of a color wheel, a plurality of redgreen blue sliders, a plurality of Hue Saturation Value sliders, and acolor temperature slider.

In various implementations, process 1100 includes generating a fifth setof user interface elements configured to receive a fifth selection of atleast one of an intensity, an exposure, a softness, and a roundness ofthe virtual stage light.

In various implementations, process 1100 includes generating a pluralityof software switches, the software switches comprising at least one ofan active switch, an order switch, and a wrapping switch.

In various implementations, process 1100 includes generating a pluralityof software switches, the plurality of software switches comprising atleast one of a show border switch, a color and opacity switch, a displaytransform switch, and a gamut transformation switch.

Process 1100 may include additional implementations, such as any singleimplementation or any combination of implementations described belowand/or in connection with one or more other processes describedelsewhere herein.

Although FIG. 11 shows example blocks of process 1100, in someimplementations, process 1100 may include additional blocks, fewerblocks, different blocks, or differently arranged blocks than thosedepicted in FIG. 11 . Additionally, or alternatively, two or more of theblocks of process 1100 may be performed in parallel.

In various embodiments, a computing device can include one or morememories; and one or more processors in communication with the one ormore memories and configured to execute instructions stored in the oneor more memories to performing operations of any of the methodsdescribed above.

In various embodiments, a computer-readable medium storing a pluralityof instructions that, when executed by one or more processors of anelectronic device, cause the one or more processors to performoperations of any of the methods described above.

FIG. 12 is a flowchart of an example process 1200 associated withgeneration of a graphical user interface for color and lightingadjustment for immersive content production system. In someimplementations, one or more process blocks of FIG. 12 may be performedby a computing device (e.g., computing device 3100). In someimplementations, one or more process blocks of FIG. 12 may be performedby another device or a group of devices separate from or including thecomputing device. Additionally, or alternatively, one or more processblocks of FIG. 10 may be performed by one or more components of device3100, such as processing unit 3104, storage subsystem 3118, systemmemory 3110, cameras 3134, displays 3132, communications subsystems3124, and bus 3102.

At block 1210, process 1200 may include generating a first set of userinterface elements configured to receive a first selection of an offsetvalue for adjusting a color of a virtual object. For example, thecomputing device may generate a first set of user interface elementsconfigured to receive a first selection of an offset value for adjustinga color of a virtual object, as described above. The first selection canbe received via a GUI on a tablet computing device as described below.The first selection can be received via a light capture device asdescribed below. The first selection can be received via an input/outputdevice on a computing device (e.g., a desktop or laptop computer). Thefirst selection can be received via a virtual interface using augmentedreality devices or virtual reality devices.

In a various implementations, at least one of the first, the second, orthe third set of user interface elements comprise at least one of acolor wheel, a plurality of Red Green Blue sliders, a plurality of HueSaturation Value sliders, and a color temperature slider.

At block 1220, process 1200 may include generating a second set of userinterface elements configured to receive a second selection of a gammavalue for adjusting the color of the virtual object. For example, thecomputing device may generate a second set of user interface elementsconfigured to receive a second selection of a gamma value for adjustingthe color of the virtual object, as described above. The secondselection can be received via a GUI on a tablet computing device asdescribed below. The second selection can be received via a lightcapture device as described below. The second selection can be receivedvia an input/output device on a computing device (e.g., a desktop orlaptop computer). The second selection can be received via a virtualinterface using augmented reality devices or virtual reality devices.

At block 1230, process 1200 may include generating a third set of userinterface elements configured to receive a third selection of a gainvalue for adjusting the color of the virtual object. For example, thecomputing device may generate a third set of user interface elementsconfigured to receive a third selection of a gain value for adjustingthe color of the virtual object, as described above. The third selectioncan be received via a GUI on a tablet computing device as describedbelow. The third selection can be received via a light capture device asdescribed below. The third selection can be received via an input/outputdevice on a computing device (e.g., a desktop or laptop computer). Thethird selection can be received via a virtual interface using augmentedreality devices or virtual reality devices.

At block 1240, process 1200 may include generating a software switch forreceiving a fourth selection to enable or disable the offset value, thegamma value, and the gain value adjustments to the color of the virtualobject (block 1240). For example, the computing device may generate asoftware switch for receiving a selection to enable or disable theoffset value, the gamma value, and the gain value adjustments to thecolor of the virtual object, as described above. The fourth selectioncan be received via a GUI on a tablet computing device as describedbelow. The fourth selection can be received via a light capture deviceas described below. The fourth selection can be received via aninput/output device on a computing device (e.g., a desktop or laptopcomputer). The fourth selection can be received via a virtual interfaceusing augmented reality devices or virtual reality devices.

In various implementations, process 1200 includes generating a fourthset of user interface elements configured to receive a fourth selectionof at least one of an exposure, a saturation, and a contrast of thevirtual object.

In various implementations, process 1200 includes generating a fifth setof user interface elements configured to receive a fifth selection of atleast one of a mix, a softness, and a roundness of the virtual object.

In various implementations, process 1200 includes generating a pluralityof software switches, the plurality of software switches comprising atleast one of a bounds switch, an all projectors switch, and an allcameras switch.

In various implementations, process 1200 includes generating abrightness slider for at least one of the first set of user interfaceelements, the second set of user interface elements, and the third setof user interface elements.

Process 1200 may include additional implementations, such as any singleimplementation or any combination of implementations described belowand/or in connection with one or more other processes describedelsewhere herein.

Although FIG. 12 shows example blocks of process 1200, in someimplementations, process 1200 may include additional blocks, fewerblocks, different blocks, or differently arranged blocks than thosedepicted in FIG. 12 . Additionally, or alternatively, two or more of theblocks of process 1200 may be performed in parallel.

In various embodiments, a computing device can include one or morememories; and one or more processors in communication with the one ormore memories and configured to execute instructions stored in the oneor more memories to performing operations of any of the methodsdescribed above.

In various embodiments, a computer-readable medium storing a pluralityof instructions that, when executed by one or more processors of anelectronic device, cause the one or more processors to performoperations of any of the methods described above.

FIG. 13 is a flowchart of an example process 1300 associated withgeneration of a graphical user interface for color and lightingadjustment for immersive content production system. In someimplementations, one or more process blocks of FIG. 13 may be performedby a computing device (e.g., computing device 3100). In someimplementations, one or more process blocks of FIG. 13 may be performedby another device or a group of devices separate from or including thecomputing device. Additionally, or alternatively, one or more processblocks of FIG. 10 may be performed by one or more components of device3100, such as processing unit 3104, storage subsystem 3118, systemmemory 3110, cameras 3134, displays 3132, communications subsystems3124, and bus 3102.

At block 1310, process 1300 may include generating a first set of userinterface elements configured to receive a first selection of a color ofa panosphere. For example, the computing device may generate a first setof user interface elements configured to receive a first selection of acolor of a panosphere, as described above. The first selection can bereceived via a GUI on a tablet computing device as described below. Thefirst selection can be received via a light capture device as describedbelow. The first selection can be received via an input/output device ona computing device (e.g., a desktop or laptop computer). The firstselection can be received via a virtual interface using augmentedreality devices or virtual reality devices.

In various implementations, the first set of user interface elementsconfigured to receive the first selection of the color for thepanosphere comprise at least one of a color wheel, a plurality of RedGreen Blue sliders, a plurality of Hue Saturation Value sliders, and acolor temperature slider.

At block 1320, process 1300 may include generating a second set of userinterface elements configured to receive a second selection of aposition and an orientation of the panosphere. For example, thecomputing device may generate a second set of user interface elementsconfigured to receive a second selection of a position and anorientation of the panosphere, as described above. The second selectioncan be received via a GUI on a tablet computing device as describedbelow. The second selection can be received via a light capture deviceas described below. The second selection can be received via aninput/output device on a computing device (e.g., a desktop or laptopcomputer). The second selection can be received via a virtual interfaceusing augmented reality devices or virtual reality devices.

In various implementations, the second set of user interface elementsconfigured to receive the second selection of the position and anorientation of the panosphere comprise at least one of a pan wheel, atilt wheel, and a height slider.

At block 1330, process 1300 may include generating a window containingpreview the panosphere. For example, the computing device may generate awindow containing preview the panosphere, as described above.

In various implementations, process 1300 includes generating a third setof user interface elements configured to receive a third selection of atleast one of an exposure, a saturation, and a contrast of thepanosphere.

In various implementations, process 1300 includes generating a switch toreceive a selection to invert movement of direction of at least one ofsecond set of user interface elements for the panosphere.

In various implementations, process 1300 includes generating a switch toreceive a selection for a fade to gray transition effect of thepanosphere.

In various implementations, process 1300 includes generating a softwareswitch for receiving a selection to enable or disable adjustments to thepanosphere.

Process 1300 may include additional implementations, such as any singleimplementation or any combination of implementations described belowand/or in connection with one or more other processes describedelsewhere herein.

Although FIG. 13 shows example blocks of process 1300, in someimplementations, process 1300 may include additional blocks, fewerblocks, different blocks, or differently arranged blocks than thosedepicted in FIG. 13 . Additionally, or alternatively, two or more of theblocks of process 1300 may be performed in parallel.

In various embodiments, a computing device can include one or morememories; and one or more processors in communication with the one ormore memories and configured to execute instructions stored in the oneor more memories to performing operations of any of the methodsdescribed above.

In various embodiments, a computer-readable medium storing a pluralityof instructions that, when executed by one or more processors of anelectronic device, cause the one or more processors to performoperations of any of the methods described above.

VI. Color Correction Windows

FIG. 14 is a flowchart of an example process 1400 associated with coloradjustments for immersive content production system. A color correctionwindow can be used for a color correction edit (e.g., a gamma, gain,exposure) on a 2D region. In some implementations, one or more processblocks of FIG. 14 may be performed by a computing device (e.g.,computing device 3100). In some implementations, one or more processblocks of FIG. 14 may be performed by another device or a group ofdevices separate from or including the computing device. Additionally,or alternatively, one or more process blocks of FIG. 10 may be performedby one or more components of device 3100, such as processing unit 3104,storage subsystem 3118, system memory 3110, cameras 3134, displays 3132,communications subsystems 3124, and bus 3102.

At block 1410, process 1400 may include identifying a color mismatchbetween a portion of the performer or the physical object and a virtualimage of the performer or the physical object in the images of thevirtual environment. For example, the computing device may identify acolor mismatch between a portion of the performer or the physical objectand a virtual image of the performer or the physical object in theimages of the virtual environment, as described above.

At block 1420, process 1400 may include capturing a plurality of imagesof a performer or a physical object in the performance area along withat least some portion of the images of the virtual environment by ataking camera. For example, the computing device may identify a colormismatch between a portion of the performer or the physical object and avirtual image of the performer or the physical object in the images ofthe virtual environment, as described above.

At block 1430, process 1400 may include identifying a color mismatchbetween a portion of the performer or the physical object and a virtualimage of the performer or the physical object in the images of thevirtual environment. For example, the computing device may identify acolor mismatch between a portion of the performer or the physical objectand a virtual image of the performer or the physical object in theimages of the virtual environment.

At block 1440, process 1400 may include generating a patch for theimages of the virtual environment to correct the color mismatch. Forexample, the computing device may generate a patch for the images of thevirtual environment to correct the color mismatch.

At block 1450, process 1400 may include inserting the patch into theimages of the virtual environment. For example, the computing device mayinsert the patch into the images of the virtual environment.

At block 1460, process 1400 may include generating content based on theplurality of captured images. For example, the computing device maygenerate content based on the plurality of captured images.

In a first implementation, the patch is displayed flat with respect tothe one or more LED displays.

In various implementations, process 1400 includes adjusting an intensityof the patch displayed on the one or more LED displays.

In various implementations, process 1400 includes adjusting a positionof the patch displayed on the one or more LED displays.

In various implementations, process 1400 includes adjusting anorientation of the patch displayed on the one or more LED displays.

In various implementations, process 1400 includes adjusting an exposureof the patch displayed on the one or more LED displays.

In various implementations, process 1400 includes adjusting a size ofthe patch displayed on the one or more LED displays.

Process 1400 may include additional implementations, such as any singleimplementation or any combination of implementations described belowand/or in connection with one or more other processes describedelsewhere herein.

Although FIG. 14 shows example blocks of process 1400, in someimplementations, process 1400 may include additional blocks, fewerblocks, different blocks, or differently arranged blocks than thosedepicted in FIG. 14 . Additionally, or alternatively, two or more of theblocks of process 1400 may be performed in parallel.

In various embodiments, a computing device can include one or morememories; and one or more processors in communication with the one ormore memories and configured to execute instructions stored in the oneor more memories to performing operations of any of the methodsdescribed above.

In various embodiments, a computer-readable medium storing a pluralityof instructions that, when executed by one or more processors of anelectronic device, cause the one or more processors to performoperations of any of the methods described above.

FIG. 15 illustrates a track pad user interface 1500. The track pad userinterface 1500 can be execute on a computing device (e.g., a tabletcomputer or a notebook computer). The track pad user interface 1500connects with the immersive content production system 900 via a wirelessprotocol (e.g., Bluetooth, Bluetooth Low Energy, Wi-Fi, etc.).

The enabled switch 1502 can turn the functions of the user interface onand off. The enabled switch 1502 can turns on/off a reticle or pointeron the immersive content display screens. As shown in FIG. 15 , theenabled switch 1502 has been selected to an “ON” position.

The point space button 1506 can reconfigure the drawing area 1504depending on the pointer area. The pointer space button 1506 can allowfor selection of wall mode using a wall button 1506 or ceiling modeusing a ceiling button 1508. The wall mode refers to projection of thepointer on the semicircular LED wall area as described above. Theceiling mode refers to projection of the pointer on the LED displaysmounted on the ceiling in the performance area of the immersive contentproduction system. As shown in FIG. 15 , the wall mode button 1506 hasbeen selected. In various embodiments, a floor mode button 1508 can beselected.

The pointer mode buttons can allow an operator to select an absolutemode button 1512 or a relative mode button 1514 to select the coordinatesystem used by the pointer. By selecting the absolute mode button 1512,the coordinates of the drawing area 1504 represents the entire wall areain the performance area of the immersive content production system. InFIG. 15 , the absolute mode button 1512 is selected. The relative modebutton 1514 enable a relative mode for the pointer and can allow forzooming into a particular area of the displays to provide for greaterdetail. A reset mode button 1518 can reset the track pad user interface1500 to a default setting upon selection.

The absolute coordinate system 1514 can be labelled from 0 degrees to165 degrees to the right of a center of the display and 0 degrees to 195degrees to the left of the center of the display. The left and rightedges of the drawing area 1504 represent 180 degrees absolute.

The relative coordinate system 1516 can be labeled 12 o'clock to 5:30o'clock (to the right of the center of the display) and 12 o'clock to6:30 o'clock to the left of center of the display. The left and rightedges of the drawing area 1504 represent 6 o'clock relative.

The point color button 1520 allows for changing the color of the pointeron the display. A pointer's enabled button 1522 allows for enabling ordisabling of all pointers on the displays. The recent selection button1524 allows preservation of one or more objects in the display that havebeen recently (e.g., within a predetermined time period) tapped on bythe user. FIG. 15 indicates no recent selections 1526. The point atcurrent selection button 1528 allows for moving the pointer to an areadesignated by one or more other controllers for the immersive contentgeneration system (e.g., a central control table).

The selection mode buttons allows provide a set (“equals”) mode button1530 to select virtual objects within the 3D scene by tapping inside ofthe drawing area 1504. An addition (“union”) button 1532 can allow foradding one or more additional object(s) from the 3D scene to the globallist of selected scene elements/objects. A subtraction mode button 1534allows the operator to remove scene elements from the globally-selectedscene elements/objects. The highlight disabled button 1536 allows fordisabling the highlighted areas in the drawings area 1504. The zoomfactor button 1538 allows for selection of a zoom factors from aplurality of zoom factors for the drawing area 1504. In FIG. 15 , theselected zoom factor is 1.0.

FIG. 16 illustrates a stage positioning panel 1600. The stagepositioning panel 1600 can be used to designate a center point 1602(e.g., an origin or motion capture stage origin point). The center point1602 is a point in the performance area from the virtual scene isprojected from. This can be a central location in the virtual scene. Itmay be desirable to relocate the center point 1602 due to filmingrequirements to another location in the performance area. For example,the starting position for characters in the virtual scene can be movedduring filming. The stage positioning panel 1600 can be used for stagealignment to meet the director preferences.

The stage positioning panel 1600 can provide location information 1604(e.g., in X, Y, Z coordinates) for the center point 1602. A reset-Xbutton can reset the X position for the center point 1602 to a default Xvalue. A reset-Y button 1608 can reset the Y position for the centerpoint 1602 to a default Y value. The reset-Z button 1610 can reset the Zposition for the center point 1602 to a default Z value. The reset allbutton 1612 can reset the X position, the Y position, and the Z positionof the center point 1602 to a default X, Y, and Z values.

A rotation value 1614 allows for rotating the scene at the center point1602 by a number of degrees by an operator entering a number (e.g., 0 to360 degrees). A virtual rotation dial 1616 allows a user to turn thevirtual rotation dial 1616 via a touchscreen interface to rotate thescene at the center point 1602. The sensitivity and scale of the virtualrotation dial 1616 can be adjusted to allow for fine or coarseadjustment.

A Move-Y selector 1618 can be a virtual slider switch that can allow auser to fine tune the Y value of the center point 1602 by moving theslider bar. A Move-XY selector 1618 can be a virtual controller to moveand fine tune the position of the center point 1602 by moving thevirtual controller via a touch interface.

The stop virtual button 1622 can cease the movement of the center point1602. The movement scale can be adjusted using a movement scale virtualslider 1624. The movement scale value 1626 can be displayed on the stagepositioning panel 1600.

A virtual slider bar 1628 can provide for adjustment of the position ofthe center point 1602. The fine-tuned Y slider 1630 can allow forfine-tuned adjustments of the Y-values for the center point 1602 by ausing applying touch movements along the fine-tuned Y slider 1630. Thefine-tuned X slider 1632 can allow for fine-tuned adjustments of theX-values for the center point 1602 by a using applying touch movementsalong the fine-tuned X slider 1632. The fine-tuned Z slider 1634 canallow for fine-tuned adjustments of the Z-values for the center point1602 by a using applying touch movements along the fine-tuned Z slider1634.

A set as origin button 1636 can allow for designating the currentposition of the center point 1602 as the new origin. This can overwritethe previously stored value for the origin. A reset to initial originbutton 1638 can allow the origin point to be reset to the previouslystored origin value. The Reset to World 1610 button can reset thecurrent position of the center point 1602 to the 3D scene's world originvalue, typically (0.0, 0.0, 0.0).

FIG. 17 illustrates an animation triggers panel 1700. Operators canprogram a variety of animations for displaying on the immersive contentdisplay panels. The animations can include explosions, weather events(e.g., tornadoes, lightning), fireworks, a starship jumping tohyperspace, etc. The animations can be stored in the immersive contentproduction system to be triggered at the right point in time which canbe synchronized with the live actors in the performance area. Theanimations can allow for the effects to be more dynamic and meet thedynamic needs of a director's requests for the actors' performance. Thetriggering of the animations may not always occur at a fixed point intime because often the timing can depend on actions or cues from humanactors.

The animation triggers can initiate changes in lighting effects and notjust visual effects. The animation triggers panel 1700 can be used forre-timing animations.

The animation triggers panel 1700 can allow an operator to select one ormore pre-programmed animations. The operator can depress a launch button1702 at the right point in time to trigger the selected animation. Theoperator can depress a stop button 1704 to end the selected animation. Areset button 1706 can restore the animations to a start of the action.

The animations triggers panel 1700 can include many different animationsthat can be added using an addition button 1708 and selected by theoperator. The animations triggers panel 1700 can be adjustable to listthe animations in a sequence as desired by the operator.

FIG. 18 illustrates a virtual stage lights panel 1800. The operator canuse the virtual stage lights panel 1800 to generate and adjust one ormore virtual stage lights as discussed above. A virtual stage light caninclude at least three overarching features: a shape 1802, an image1804, and a gradient 1806. In various embodiments, the shape 1802 of thevirtual stage lights can be one of a square, a rectangle, a circle, arounded square, or a triangle. Other shapes can be selected. The shape1802 can define the outline of the virtual stage light.

The operator can select an image 1804 for the virtual stage light. Thevirtual stage lights can be drawn to simulate various different images(e.g., a round light, a group of lights, or the light from the moon).The operator can select one of a plurality of images 1804 of variouslights to map onto the selected shape 1802. If the no image indicator1806 is selected the virtual stage light can appear as the shape 1802that has been selected (e.g., a rectangle, circle, triangle, etc.) Forexample, by mapping an image 1804 onto a shape 1802, the appearance of avirtual stage light can replicate the selected image. A street lampvirtual stage light can be mapped onto an image 1804 that replicates astreet lamp. The virtual stage lights panel 1800 can provide gradientcontrols 1806 for defining an edge of the virtual stage light (e.g., theedge can be sharp or blurred).

The virtual stage lights panel 1800 can provide controls for varying thelocation and orientation of the virtual stage lights on the displays. Arotation value 1808 can indicate the current rotation value for thevirtual stage lights. The rotation controller 1810 can allow an operatorto adjust the rotation value through the virtual adjustment wheel. Thelatitude value 1812 value and the longitude value 1814 of a center ofthe virtual stage light can be displayed on the virtual stage lightspanel 1800. The latitude position of the center of the virtual stagelight can be modified using a coarse virtual latitude slider 1816. Alatitude position fine-tune controller 1818 can be used to make minoradjustments to the latitude value 1812 for the location of the center ofthe virtual stage light. A longitude position controller 1820 can beused to adjust a position of the center of the virtual stage light. Afine-tuned controller can also be provided for minor adjustments in thelongitude of the position of the center of the virtual stage light 1800.

Each virtual stage light can have various size properties defined byusing the virtual stage lights panel 1800. A height indicator 1822 candefine a height of a virtual stage light. A width indicator 1824 candefine a width of the virtual stage light. An aspect ratio lock 1823 canbe used to lock the aspect ratio of width to height. With the aspectratio lock 1823 engaged adjusting either the height or width of thevirtual stage light will adjust the other value to maintain the aspectratio of the virtual stage light.

A color wheel 1826 can allow the operator to select a color value forthe virtual stage light. The operator can select a particular color onthe color wheel. Alternatively, the operator can select a Hue value1828, a saturation value 1830, and a luminosity value 1832 (HSV) fromthe HSV Color Scale. The HSV scale provides a numerical readout of aselected color that corresponds to the color names contained therein.Hue can be measured in degrees from 0 to 360. For instance, cyan fallsbetween 181-240 degrees, and magenta falls between 301-360 degrees. Theluminosity value and saturation of a color are both analyzed on a scaleof 0 to 100 percent. Most digital color selection interfaces can bebased on the HSV scale, and HSV color models are particularly useful forselecting precise colors for virtual stage lights in the display.

Alternatively, the color for the virtual stage lights can be definedusing a red, green, blue (RGB) color model. The RGB color model can bebased on the color theory that all visible colors can be made using theadditive primary colors of red, green, and blue. As the operator adjuststhe amount these basic colors, you can create a variety of differentcolors. The red value 1834, green value 1836, and blue value 1838 can bepresented on the virtual stage lights panel 1800 for the selected color.

An opacity slider 1840 can be used to adjust the alpha channel (e.g.,how opaque versus transparent) that Lightcard itself is. A gradientscale 1840 illustrates a fine tuned color for a selected color value1842. A color picker 1844 (e.g., eye dropper icon) allows an operator tomatch an existing color from other virtual objects for use for a virtualstage light.

A color temperature selector 1846 allows an operator to select aspecific color temperature value of the base color (or white color).Color temperature can be described as warmer (orange) or cooler (blue).Color temperature is a way to describe the light appearance provided bya light bulb. It is measured in degrees of Kelvin (K) on a scale from1,000 to 12,000. A color temperature value 1848 (e.g., 6500 Kelvin) canbe entered by the operator. A color temperature slider 1850 can allowfor fine tuning the specific color temperature value 1848. The colortemperature slider 1850 can be move to the left for warmer values and tothe right to colder values.

All of the different ways to change or select a color of the tint of thevirtual stage light can be connected together. Therefore, if an operatorselects a color from the color wheel 1826 based on the position of thecolor selector 1842, the red value 1834, green value 1836, and bluevalue 1838 can be changed based on the selection. In addition, the Huevalue 1828, a saturation value 1830, and a luminosity value 1832 (HSV)from the HSV Color Scale will change for the selection. Similarlychanging the HSV values can change the red value 1834, green value 1836,and blue value 1838 displayed and the position of the color selection1842.

The virtual stage lights panel 1800 can allow the operator to select anintensity value 1852, an exposure value 1854, a softness value 1856, anda roundness value 1858 for virtual stage lights. The intensity value1852 can be a luminous intensity that can be measured in lumens persteradian or candela. The exposure value 1854 defines an amount of lightin a scene. The softness value 1856 is another property of light. Softlight is light that tends to “wrap” around objects, projecting diffusedshadows with soft edges, whereas hard light is more focused and producesharsher shadows. The hardness or softness of light depends mostly onthree features of the source: the size of its surface, its distance fromthe object, and the thickness of its diffusion material. A large,distant light source with thick diffusion material will produce softerlighting than one that is smaller and closer to the subject, withthinner diffusion material. Here, Softness and Roundness can be used tospecify the edge of the Lightcard “shape”—softness larger means it is amore “feathered” edge.

The roundness value 1858 is another property of the virtual stagelights. The roundness value 1858 changes the shape of the light from asquare at 0, to rounded corners, to a disk at 1.

The active switch 1860 allows an operator to turn on and off the virtualstage light. FIG. 18 illustrates the active switch 1860 in the “ON”position. The wrapping switch 1862 allows an operator to wrap thevirtual stage light between different displays, modifying how theLightcard is projected onto the display surfaces. The show border switch1865 displays a border around the Lightcard on the displays. The colorand opacity selector 1866 can allow the Lightcard to display an imageusing either RGB or full-RGBA properties. The transform switch 1868 canapply a display transform to the virtual stage light. The gamuttransform switch 1870 applies a gamut transformation to the virtualstage light. Gamut mapping is the problem of transforming the colors ofimage or video content so as to fully exploit the color palette of thedisplay device. Gamut and Display Transform can be related to whetherthe operator wants the Lightcard (or Sticker) to be affected byscene-level color correction or not, and if their values should changewhen the “Color Pipeline” changes. For example, if the camera operatorputs a different lens on the taking/shooting camera, the color pipelinecan change and the way that colors are interpreted needs to change toreflect the needs of the new lens. The question for Lightcards thenbecomes, “Should this lightcard change to reflect the change in lens? Orshould it stay the same?”

A Lightcard, like a “real life light”, does not change itself when thecolor pipeline changes. Imagine a real-life lightbulb on set—if a setcrew changes what is happening inside of the stage, the costumes, thebackgrounds the real-life light does not change and it just keepsshining at the same brightness and the same intensity and color etc.Lightcards are not affected by the color pipeline changes (lens changes,camera body changes, etc.)

A Sticker, on the other hand, can often be used to patch up the scene orsomething else, and therefore it is used to ‘blend’ elements in the 3Dscene. When the lens changes and color interpretation changes as aresult it can be desirable for stickers to have those “Color Transforms”applied.

So, Gamut and Display Transform can be used to be specifiableindividually. In various embodiment, an operator may choose either aSticker or a Lightcard and the Gamut/Display decision can be madeinternally depending on the selection. The selection at creation-timeimplies an expected workflow if something triggers a change in the colorpipeline (e.g., a change in lens or camera body) during a shoot. Thesort order selector 1872 can control the “Z-depth” of the variousLightcards so the operator can control which Lightcard drawings on topof other Lightcards that overlap.

FIG. 19 illustrates a second virtual stage lights/sticker panel 1900.FIG. 19 has similar functions as virtual stage lights panel 1600,described for FIG. 16 , but has the additional functionality foradjusting an appearance and location of 2-D virtual objects (e.g.,stickers). The operator can use the virtual stage lights/sticker panel1900 to generate and adjust one or more virtual stage lights and/orstickers. The virtual stage light/sticker panel 1900 can include tabs toselect features of a selected virtual stage light and/or a selectedsticker. The tabs can include a shape tab 1902, an image tab 1904, and agradient tab 1906. As illustrated in FIG. 19 , the shape tab 1902 isselected. In various embodiments, the shape of the virtual stage lightand/or sticker can be one of a circle 1908, a square 1910, or a triangle1912. Other shapes can be included. The shape tab 1902 can define theoutline of the virtual stage light and/or sticker.

Using the image tab 1904, the operator can select one of a plurality ofimages 1904, as shown in FIG. 18 , of various lights to map onto theshape. A gradient tab 1906 can allow the operator to adjust the edge ofthe virtual stage light and/or sticker (e.g., the edge can be sharp orblurred).

The virtual stage lights/sticker panel 1900 can provide controls forvarying the location and orientation of the virtual stage light and/orsticker on the displays. A longitude position indicator 1914 can presentthe current longitude position of a center of the virtual stage lightand/or sticker on the display. The operator can adjust the longitudeposition of the virtual stage light and/or sticker using the virtuallongitude controller dial 1916. A fine-tuned controller can also beprovided for minor adjustments in the longitude of the position of thecenter of the virtual stage light and/or sticker.

Each virtual stage light and/or sticker can have various size propertiesdefined by using the virtual stage lights/sticker panel 1900. A widthslider 1918 can be used to adjust the width of the virtual stage lightand/or sticker. A height slider 1920 can adjust the height of thevirtual stage light and/or sticker. An aspect ratio lock 1922 can beused to lock the aspect ratio of width to height. With the aspect ratiolock 1922 engaged adjusting either the height or width of the virtualstage light and/or sticker will adjust the other value to maintain theaspect ratio of the virtual stage light and/or sticker.

The latitude position indicator 1924 can indicate a current latitudeposition of the center of the virtual stage light and/or sticker on thedisplay. The latitude position of the virtual stage light and/or stickercan be modified using a coarse virtual latitude slider 1926. A latitudeposition fine-tune controller 1928 can be used to make minor adjustmentsto latitude for a position of the center of the virtual stage lightand/or sticker.

A rotation value 1930 can indicate the current rotation value for thevirtual stage light and/or sticker. The rotation controller 1932 canallow an operator to adjust the rotation value 1930 through the virtualadjustment wheel 1932.

A light illuminant portion of the virtual stage light/sticker panel 1900can allow for adjustment of a color for the virtual stage light and/orsticker. A white/neutral button 1934, a temperature button 1936, and achromaticity button 1938 can be presented on the virtual stage lightpanel 1900.

A color wheel 1940 can allow an operator to define a color value for thevirtual stage light and/or sticker. The operator can select a particularcolor selection 1942 on the color wheel. Alternatively, the operator canselect a Hue value 1944, a saturation value 1946, and a luminosity value1948 from the HSV Color Scale. The HSV scale provides a numericalreadout of a selected color that corresponds to the color namescontained therein.

Alternatively, the color for the virtual stage light and/or sticker canbe defined using a red, green, blue (RGB) color model. The RGB colormodel can be based on the color theory that all visible colors can bemade using the additive primary colors of red, green, and blue. The redvalue 1950, green value 1952, and blue value 1954 can be presented onthe display for the selected color 1942.

An opacity slider 1840 can be used to adjust the alpha channel (e.g.,how opaque versus transparent) that Lightcard or Sticker is for aselected color value 1942. A color picker 1958 (represented by an eyedropper) allows an operator to match an existing color for use for avirtual stage light and/or sticker.

A color temperature selector 1942 allows an operator to select aspecific color temperature value of the base color. A color temperaturevalue 1962 can be entered by an operator. A color temperature slider1964 can allow for fine tuning the specific color temperature value1962.

All of the different ways to change the color or the tint of the virtualstage light are connected together. Therefore, if an operator selects acolor from the color wheel, the red value 1950, green value 1952, andblue value 1954 will be represented for the selection. In addition, theHue value 1944, a saturation value 1946, and a luminosity value 1948(HSV) from the HSV Color Scale will change for the selection.

The virtual stage lights/sticker panel 1900 can allow the operator toselect an intensity value 1966, an exposure value 1968, a softness value1970, and a roundness value 1972 for virtual stage lights. The intensityvalue 1966 can be a luminous intensity that can be measured in lumensper steradian or candela. The exposure value 1968 defines an amount oflight in a scene. The softness value 1970 is another property of light.Soft light is light that tends to “wrap” around objects, projectingdiffused shadows with soft edges, whereas hard light is more focused andproduces harsher shadows. The hardness or softness of light dependsmostly on three features of the source: the size of its surface, itsdistance from the object, and the thickness of its diffusion material. Alarge, distant light source with thick diffusion material will producesofter lighting than one that is smaller and closer to the subject, withthinner diffusion material.

The roundness value 1972 is another property of the virtual stagelights. The roundness value 1972 changes the shape of the light from asquare at 0, to rounded corners, to a disk at 1.

The active switch 1974 allows an operator to turn on and off the virtualstage light and/or sticker. The wrapping switch 1976 allows an operatorto wrap the virtual stage light and/or sticker between differentdisplays. The show border switch 1978 allows for showing the border. Thecolor and opacity selector 1980 can allow the operator to select colorand opacity or opacity only. The transform switch 1982 can apply adisplay transform to the virtual stage light and/or sticker. The gamuttransform switch 1984 applies a gamut transformation to the virtualstage light and/or sticker. Gamut mapping is the problem of transformingthe colors of image or video content so as to fully exploit the colorpalette of the display device. The sort order selector 1886 can controlthe “Z-depth” of the various Lightcards/Stickers so the operator cancontrol which Lightcard/Stickers drawings on top of otherLightcards/Stickers that overlap.

FIG. 20 illustrates a first panosphere control panel 2000 forcontrolling the appearance of a sky portion of a three-dimension sceneof the immersive content generation system. The panosphere can be apanoramic photo stitched together from a single point in everydirection. The panosphere can be a portion of the sky that can bedefined by latitude/longitude coordinates. As an example, a panospherecan be used for a virtual reality headset in looking at the sky theimage is replicated for 360 degrees around. These images can be capturedat various locations around the world using spherical cameras andconnected together. In general, panospheres are used to depict elementsthat are a far distance away (e.g., infinite distance). In variousembodiments the panosphere can include multiple panosphere layers put ontop of each other (e.g., a haze pass, a background, clouds, otherplanets, stars). These other elements can be broken up into multiplepanosphere layers and presented together in a certain order. In variousembodiments, the different layers can move independent of each other(e.g., movement of planets across the sky).

The operator can adjust the color, position, and lightningcharacteristics of the panosphere in a scene using the controls on thepanosphere control panel 2000. The panosphere control panel 2000 caninclude many of the same lighting controls as shown for FIGS. 18 and 19.

The operator can create a new panosphere image file or load a panosphereimage file from memory. The name 2002 of the panosphere image file canbe displayed.

The operator can adjust the exposure of the panosphere using theexposure slider 2004. The exposure value 2006 can be displayed.

The operator can adjust the saturation of the panosphere using thesaturation slider 2008. The saturation value 2010 can be displayed.

The operator can adjust the contrast of the panosphere using thecontrast slider 2012. The contrast value 2014 can be displayed.

The operator can adjust the gamma of the panosphere using the gammaslider 2016. The gamma value 2018 can be displayed.

The color gain can be adjusted via a plurality of sliders. The operatorcan adjust the red color value 2020 of the panosphere using the redcolor slider 2022. The operator can adjust the green color value 2024 ofthe panosphere using the green color slider 2026. The operator canadjust the blue color value 2028 of the panosphere using the blue colorslider 2030. The operator can adjust the temperature of the panosphereusing the temperature slider 2032.

The operator can rotate the panosphere around a center point using thepan wheel 2034. The operator can adjust the tilt of the panosphere usingthe tilt wheel 2036.

The operator can adjust the height of the panosphere using a coarseheight adjustment slider 2038 and a fine height adjustment slider 2040.The height value 2042 can be presented on the display. The operator caninvert the movement direction using the invert switch 2044.

A first preview box 2046 can display a preview of a first scene. Asecond preview box 2048 can display a preview of a second scene. A thirdpreview box 2050 can display a preview of a third scene. The firstpreview box 2046, the second preview box 2048, and third preview box2050 provide static and/or animated backgrounds that can be loaded intothe immersive content generation system.

A focus edit button 2052 can allow for editing the focus point of thedisplay. The focus view button can display the focus point on thedisplay. These buttons scroll the “focus” of the scroll list on the lefthand side (that contains all the possible Panosphere images that can beloaded). An operator can scroll the list to the currently-selected panoimage—either the one that is currently being edited or the one that iscurrently being “viewed” (aka loaded on the walls). It is possible thatthe currently-being-edited and currently-being-viewed pano selection isthe same image (shown in a row in the list of selectable images) orcould be different ones. The preload all button 2056 allows variouspanosphere images to be preloaded. The active switch 2058 allows thepanosphere images to activated or de-activated on the immersive contentdisplays. The draw order selector 2060 allows for designation of theorder for various images of the panosphere. The panosphere image 2062displays the color of the panosphere. The fade-to-gray image transitionswitch 2064 allows for toggling the fading feature on and off.

FIG. 21 illustrates a second panosphere control panel 2100 forcontrolling the appearance of a sky portion of a three-dimension sceneof the immersive content generation system. The panosphere is just apanoramic photo stitched together from a single point in everydirection. The panosphere can be a portion of the sky that is defined bylatitude/longitude coordinates. As an example, a panosphere can be usedfor a virtual reality headset in looking at the sky the image isreplicated for 360-degrees around. These images can be captured atvarious locations around the world using spherical cameras and connectedtogether.

The operator can adjust the color, position, and lightingcharacteristics of the panosphere in a scene using the controls on thepanosphere control panel 2100. The panosphere control panel 2100 caninclude many of the same lighting controls as shown for FIGS. 16 and 17.

The operator can create a new panosphere image file or load a panosphereimage file from memory. The name 2102 of the panosphere image file canbe displayed.

The operator can adjust the exposure of the panosphere using theexposure slider 2104. The exposure value 2106 can be displayed.

The operator can adjust the saturation of the panosphere using thesaturation slider 2108. The saturation value 2110 can be displayed.

The operator can adjust the contrast of the panosphere using thecontrast slider 2112. The contrast value 2114 can be displayed.

The operator can adjust the gamma of the panosphere using the gammaslider 2116. The gamma value 2118 can be displayed.

The color gain can be adjusted via a plurality of sliders. The operatorcan adjust the red color value 2120 of the panosphere using the redcolor slider 2122. The operator can adjust the green color value 2124 ofthe panosphere using the green color slider 2126. The operator canadjust the blue color value 2128 of the panosphere using the blue colorslider 2130. The operator can adjust the temperature of the panosphereusing the temperature slider 2132.

The operator can rotate the panosphere around a center point using thepan wheel 2134. The operator can adjust the tilt of the panosphere usingthe tilt wheel 2136.

The operator can adjust the height of the panosphere using a coarseheight adjustment slider 2138 and a fine height adjustment slider 2140.The height value 2142 can be presented on the display. The operator caninvert the movement direction using the invert switch 2144.

The A/B and A→B switches can be used for comparing a set of values thean operator has chosen to an alternate set of values. The same A/Bswitcher can be used on the Color Correction panel as well. The setup isthat the operator will get some selection of settings in a way they likeit—and then then want to experiment with changing some subset of theedited values, whilst leaving some others the same. They then want tovisually compare the two buckets of settings. In effect “they want to‘A/B’ it”.

For example, an operator can dial in a few values and then the operatorcan think perhaps both exposure should go up 2 stops and saturationshould maybe also go down but a couple of units. The current switch A orB is auto-set to the existing values. The operator can then use A→B toPUSH those current values of all sliders/dials to the “B” set. Thiswould make switching between A and B have no effect since the twocontain identical settings. Then while on “B”, they will ‘stop up’exposure a bit and dial down saturation, say. The operator can then,with a single button, swap all the controls back to the previously saved“A” state, and then back to B, back to A, etc. To flick the settings alltogether back and forth, usually while looking up at the walls to judgethe result of the settings changes. Finally, the operator may decidethey prefer A or B, and will stick with that and move onto a new task,or they will adjust some more settings and continue to run comparisonsof A (or B) vs their ‘baseline’ (the opposite one, B, or A).

A first preview box 2146 can display a preview of a first scene. A focusedit button 2052 can allow for editing the focus point of the display.The focus view button 2054 can display the focus point on the display.The preload all button 2056 allows various panosphere images to bepreloaded. The active switch 2158 allows the panosphere images toactivated or de-activated on the immersive content displays. The draworder selector 2160 allows for designation of the order for variousimages of the panosphere. The panosphere image 2162 displays the colorof the panosphere. The use fade to gray image transition 2164 allows fortoggling the fading feature on and off.

FIG. 22 illustrates a detached virtual green screen panel 2200. Thevirtual green screen can be attached or detached to a drivingmotion-capture camera. The virtual green screen panel 2200 is an exampleof virtual green screen controls for a detached virtual green screen.The detached virtual green screen panel 2200 is similar to virtual stagelight controls in terms of placements. For certain captured images itmay not be possible to capture an in-camera final set of images usingthe taking camera. In these cases a virtual green screen can be used toblock images behind the actor(s) that can be replaced by computergraphics during the post-production visual effects stage of theproduction pipeline. This allows the benefits of using the benefits oflighting effects that the immersive content system provides for the restof the volume in the performance area. As a detached green screen it canbe placed somewhere in the volume similar to virtual stage lights (e.g.,moving around the space, change the height and width, turn it on or off,change the color (e.g., a green screen or blue screen)).

The detached virtual green screen panel 2200 can display the longitudevalue 2202 and latitude value 2204 of the virtual green screen. Theoperator can adjust the latitude value 2202 using the longitudeadjustment wheel 2206 and the latitude value 2204 using the latitudecoarse adjustment slider 2208 and/or the latitude fine adjustment slider2210.

The operator can adjust a width value 2212 of the green screen using awidth adjustment slider 2214. The operator can adjust a height value2216 using a height adjustment slider 2218. An aspect ratio lock 2219,if selected, the aspect ratio lock 2219 maintains an aspect ratio of thevirtual green screen. If the aspect ratio lock 2219 is selected, achange in width will result in a corresponding change in height and viceversa.

A visible switch 2220 makes the virtual visible on the displays if thevisible switch 2220 is activated. The immersive content generationsystem can generate an outline around the virtual green screen if theoutline only switch 2222 is selected. The detached virtual green screenpanel 2200 can allow an operator to select the green screen color. Invarious embodiments, a green screen button 2224, a blue screen button2226, a black screen button 2228, and a colorless or transparent “green”screen button 2230 can be selected. The detached virtual green screenpanel 2200 can present the selected color value image 2232 and anumerical color value 2234. A color value slider 2236 allows foradjustment of the color of the virtual green screen. This can be simplya 0→1 slider that controls the same value from the numerical color valueentry box (2234). The operator may want to enter an exact floating pointnumber for the Value of the color, (this is the “V” value from the “HSV”system, by the way), or the operator may just want to slide it.

Pressing color value slider 2232, it pops up a color wheel similar towhat you see on the Lightcard panel for setting the RGB/HSV with acircular color wheel type control. This affects the value as well ofcourse. The color wheel and additional “V” can be used by the operatorto dial back the intensity of the green or blue color if it is toointense of a green, for instance, and is resulting in too much brightgreen ‘light spill’ from the walls onto the actors. This can result inmore post-production visual effects work needing to happen to removethat effect.

The operator can adjust the opacity of the detached virtual green screen2200 using an opacity slider 2238.

The detached virtual green screen panel 2200 can also provide varioustools for adjusting tracking markers. Tracking markers can be “Xs” orpluses (“+”) that can be displayed on the screen. A show marker switch2240 allows the markers to be toggled on and off on the display. Theoperator can adjust the shape of the markers using the shape switch 2242to switch between an x or a plus markers. The operator can adjust thesize of the marker using the size selector 2244. The operator can adjustthe thickness of the marker using the thickness selector 2246. Theoperator can adjust a spacing of the tracking markers using the spacingadjustment 2248. The operator can adjust an offset of the variousmarkers from the virtual green screen by entering an X-offset using theoffset-X selector 2250. The operator can enter a Y-offset using theoffset-Y selector 2252.

The operator can select at least one of a green color 2254, a blue color2256, a white color 2258, and a black color 2260 for the tracker. Theoperator can adjust a value for the tracker using the value slider 2262.The detached virtual green screen panel 2200 can have preset colors forcommonly used tracker colors like black or white or green. But anoperator may want a ‘less intense green’—so rather than offering fullHSV control, which may be done in some embodiments—the detached virtualgreen screen panel 2200 can display some preset buttons here and add a“Value” only slider to reduce the V of the HSV system. This allows theoperator to dial-back the intensity as needed on the tracking markersfor the same reasons as described above for the green screen coloritself.

FIG. 23 illustrates an attached virtual green screen panel 2300. Theattached virtual green screen panel 2300 can be attached to one of themotion capture tracking cameras. The attached virtual green screen panel2300 allows the operator to change the shape of the virtual green screenwithin the frustum of the camera (so the virtual green screen may nottake up the entire viewing area). For example, box 2364 defines an areafor the virtual green screen which can be a portion of the entirefrustum 2366. The fit one-to-one button 2368 expands the box 2364 untilit matches the size of the entire frustum 2366. The attached virtualgreen screen can be attached to various cameras. For example, the cameraidentifier 2370 indicated that this attached virtual green screen isattached to camera 1. In various embodiments, there can be multipletaking cameras and there can be different attached virtual green screensattached to different taking cameras. Alternatively or additionally, theattached virtual green screens can be detached and moved around (therebybecoming a detached virtual green screen 2200 as discussed for FIG. 22).

The operator can adjust a width value 2310 of the green screen using awidth adjustment slider 2314. The operator can adjust a height value2316 using a height adjustment slider 2318. An aspect ratio lock 2318,if selected, maintains an aspect ratio of the virtual green screen. Ifthe aspect ratio lock 2318 is selected, a change in width will result ina corresponding change in height and vice versa.

A visible switch 2320 makes the virtual green screen visible on thedisplays if the visible switch 2320 is activated. The immersive contentgeneration system can generate an outline around the virtual greenscreen if the outline only switch 2321 is selected. The virtual greenscreen panel 2300 can allow an operator to select the green screencolor. In various embodiment, a green screen 2324, a blue screen 2326, ablack screen 2328, and a colorless or transparent “green” screen 2330.The virtual green screen panel 2300 can present the selected color value2332 and a numerical value 2334. A color value slider 2336 allows foradjustment of the color of the virtual green screen.

The operator can adjust the opacity of the virtual green screen using anopacity slider 2338.

The virtual green screen panel 2300 can also provide various tools foradjusting tracking markers. A show marker switch 2322 allows the markersto be toggled on and off on the display. The operator can adjust theshape (e.g., a plus sign (+) or an (x)) of the markers using the shapeswitch 2342. The operator can adjust the size of the marker using thesize selector 2344. The operator can adjust the thickness of the markerusing the thickness selector 2346. The operator can adjust a spacing ofthe tracking markers using the spacing adjustment selector 2348. Theoperator can adjust an X-offset using the offset-X selector 2350. Theoperator can adjust a Y-offset using the offset-Y selector 2352.

The operator can select at least one of a green color 2354, a blue color2356, a white color 2358, and a black color 2360 for the tracker. Theoperator can adjust a value for the tracker using the value slider 2362.The detached virtual green screen panel 2200 can have preset colors forcommonly used tracker colors like black or white or green. But anoperator may want a ‘less intense green’—so rather than offering fullHSV control, which may be done in some embodiments—the detached virtualgreen screen panel 2200 can display some preset buttons here and add a“Value” only slider to reduce the V of the HSV system. This allows theoperator to dial-back the intensity as needed on the tracking markersfor the same reasons as described above for the green screen coloritself.

FIG. 24 illustrates a color correction panel 2400. The color correctionpanel 2400 can allow for adjustment or correction of colors for virtualobjects displayed on the LED displays. This color correction can be doneto seamlessly blend the physical world (e.g., actors and props in theperformance area) and the virtual world depicted on the LED displays.The color correction allows an operator to match the colors of objectsand/or actors in the real world with depictions in the virtual world insuch a way to reduce the amount of post-production work. In this way thefilm crew can achieve as many in-camera finals as possible, therebyminimizing the costs and associated time with producing realistic visualeffects.

Color wheels are one of the most basic color correction controls. Thecolor wheels represent the color spectrum on a circle and let theoperator adjust color in the 2D space of the wheel. The color wheelsstart out with a movable dot or circle in the center of the wheel.Dragging the dot toward a color area adjusts the color of the image inthat direction. The angle at which the operator drags the moveable dotdetermines the color(s) the operator adjusts, so dragging the dot intothe red-orange area makes the overall color more red-orange. The furtheraway from the center of the wheel that the operator drags, the more theoperator increases the saturation of the color that the operator isdragging toward.

A color correction can be described as moving a dot or circle away fromthe opposite color on the wheel. So, if the selected object has an imagethat is predominately blue, dragging the dot away from the blue areareduces the amount of blue in the image. Color wheels often include aslider to control the brightness, either as a dial around the edge ofthe wheel or as a straight slider underneath it.

When using the color correction panel 2400, the operator can select avolume, a section, a patch, an object, a window, a sticker, or a portionof the display for correction. For example, the item identifier 2402lists a volume, a section, a patch, an object, a window, a sticker, or aportion of the display for correction.

In various embodiments, the color correction panel 2400 can presentthree dials to adjust the color values.

The operator can adjust the offset using the offset dial 2404. Theoffset dial 2404 can be used to controls shadows/blacks of an image. Theoffset dial 2404 can achieve this changing the brightness/color levelswhile leaving mid tones and highlight areas unaffected. The offsetselector 2406 can allow the operator to change the value of the offsetby sliding the offset selector 2406 within the offset dial 2404. Anoffset reset button 2408 sets the offset values to be reset to zero. Theoffset adjustment ring 2410 allows the operator to change the scale forthe offset values by rotating the offset adjustment ring 2410. Byrotating the offset adjustment ring 2410 the scale inside the offsetdial 2404 can change. For example, the scale can be in ones, tens,thousandths, hundredths, or hundred thousandths. The variable scaleallows for fine tuning the offset value.

Color sliders let the operator manipulate the red, green, blue, andbrightness values directly. The operator can control exactly how much ofeach color is desired. The operator can change the values of the slidersbetween using the RGB button 2412, the HSV button 2414, and the Tempbutton 2416 to allow the operator to change the slider scales. If theRGB button 2412 is selected, as shown in FIG. 24 , the first slider 2418illustrates a red value, the second slider 2420 illustrates a greenvalue, and the third slider 2422 illustrates a blue value. The offsetdial 2404 and the first slider 2418, the second slider 2420, and thethird slider 2422 are linked together so as the operator moves theoffset selector 2406 position, the first slider 2418, the second slider2420, and the third slider 2422 indicators change corresponding to theselected color adjustment.

The offset values 2424 can be shown numerically on the display. Anoffset brightness slider 2425 can be used to adjust the brightness valuefor the offset correction.

The operator can adjust the gamma using the gamma dial 2426. The gammadial 2426 can be used to control the mid tones (middle gray levels) ofan image. The color correction panel 2400 can have offset, gamma, andgain wheels affect the shadows OR the mid tones OR the highlights of theimage (OR all at once). In various embodiments, it is not the case thatoffset is only for shadows, gamma is only for midtones, and gain is onlyfor highlights. The “switcher” to choose (2468) controls which range iscurrently being edited by ALL of the controls—so if the operator chooseshadows there on the switcher, then all three of offset, gamma, and gainwheels can be used, and are frequently used, to edit the shadow ‘range’of colors in the scene. The gamma dial 2426 can achieve this by changingthe brightness/color levels of the mid tones while leaving the darks andhighlighted areas unaffected. The gamma selector 2428 can allow theoperator to change the value of the gamma by sliding the gamma selector2428 within the gamma dial 2426. A gamma reset button 2430 can reset thegamma value to zero. The gamma adjustment ring 2432 allows the operatorto change the scale for the gamma value by rotating the gamma adjustmentring 2432. By rotating the gamma adjustment ring 2432 the scale insidethe gamma dial 2426 can change. For example, the scale can be in ones,tens, thousandths, hundredths, or hundred thousandths. The variablescale allows for fine tuning the gamma value.

Color sliders let the operator manipulate the red, green, blue, andbrightness values directly. The operator can control exactly how much ofeach color is desired. The operator can change the values of the slidersbetween using the RGB button 2412, the HSV button 2414, and the Tempbutton 2416 to allow the operator to change the slider scales. If theRGB button 2412 is selected, as shown in FIG. 24 , the first slider 2418illustrates a red value, the second slider 2420 illustrates a greenvalue, and the third slider 2422 illustrates a blue value. If the HSVvalue button 2414 is selected (as shown for gamma dial 2426), the firstslider 2418 illustrates Hue value, the second slider 2420 illustratessaturation value, and the third slider 2422 illustrates luminance value.

The gamma values 2434 can be shown on the display. A gamma brightnessslider 2436 can be used to adjust the brightness value for the gammacorrection.

The operator can adjust the gain using the gain dial 2438. The gain dial2438 can be used to control the highlights or white levels of an image.In various embodiments, the gain dial 2438 can achieve this by changingthe brightness/color levels of the highlights or white levels whileleaving the darks and mid tone areas unaffected. The color correctionpanel 2400 can have offset, gamma, and gain wheels affect the shadows ORthe mid tones OR the highlights of the image (OR all at once). Invarious embodiments, it is not the case that offset is only for shadows,gamma is only for midtones, and gain is only for highlights. The“switcher” to choose (2468) controls which range is currently beingedited by ALL of the controls—so if the operator choose shadows there onthe switcher, then all three of offset, gamma, and gain wheels can beused, and are frequently used, to edit the shadow ‘range’ of colors inthe scene. The gain selector 2440 can allow the operator to change thevalue of the gain by sliding the gain selector 2440 within the gain dial2438. A gain reset button 2442 can reset the gain values to zero. Thegain adjustment ring 2444 allows the operator to change the scale forthe gain by rotating the gain adjustment ring 2444. By rotating the gainadjustment ring 2444 the scale inside the gain dial 2426 can change. Forexample, the scale can be in ones, tens, thousandths, hundredths, orhundred thousandths. The variable scale allows for fine tuning the gainvalue.

Color sliders let the operator manipulate the red, green, blue, andbrightness values directly. The operator can control exactly how much ofeach color is desired. The operator can change the values of the slidersbetween using the RGB button 2412, the HSV button 2414, and the Tempbutton 2416 to allow the operator to change the slider scales. If theRGB button 2412 is selected, as shown in FIG. 24 , the first slider 2418illustrates Red value, the second slider 2420 illustrates Green value,and the third slider 2422 illustrates Blue value. If the HSV valuebutton 2414 is selected (as shown for gamma dial 2426), the first slider2418 illustrates Hue value, the second slider 2420 illustratessaturation value, and the third slider 2422 illustrates luminance value.If the Temp button 2416 is selected (as shown for the gain dial 2438),the first slider 2418 can control a temperature value and the secondslider 2420 can control a magenta slider.

The gain values 2446 can be shown on the display. A gain brightnessslider 2448 can be used to adjust the brightness value for the gaincorrection.

The color correction active switch 2450 enables or disables the colorcorrection for the selected object. The color correction active switch2450 can be akin to visibility control so the operator can turn on oroff the state of the color correction. For example, if the operators hasa color correction value for warming up a region on the display (e.g.,making it a little bit more orange or yellow) for an exhaust pipe from amotorbike or automobile. The operator can turn it off using the colorcorrection active switch 2450 and see the scene without the results ofthe color correction enabled.

The color correction bounds switch 2452 allows the display to show thebounds of the region/object/volume in which the color correction isapplied on the display. The blend priority value 2454 allows forcontrolling the rendering priority of one color correction node overanother one (e.g., sort order button as described above).

The mix slider 2456 allows the operator to apply a global multiplier tothe color correction. A mix value of zero is essentially off value. Amix value of one means the global multiplier is fully applied. The mixslider 2456 provides a scale on the amount of how much the colorcorrection actually contributes to the total color of the selectedregion/object/volume. In effect, the mix slider 2456 provides a way toturn down the effect of this particular color correction.

The softness slider 2458 allows the operator to adjust the softness ofthe color correction.

The roundness slider 2460 allows the operator to adjust the roundness ofthe color correction.

The operator can select the region for the color correction using aMaster tab 2462, a Shadows tab 2464, a mid-tones tab 2464 and aHighlights tab 2468 and corresponding region slider 2470. By selectingthe Master tab 2462, the color correction is acting upon everything. Theshadows, mid tones, and highlights are defined by that region slider2470. On the region slider 2470, between the first dot and the seconddot is considered shadows. Between the second dot and the third isconsidered mid tones. Between the third dot and the end dot isconsidered highlights. The operator can change the position of each ofthese dots. For example, if the scene is very bright the region slider2470 can adjust the position of the dots to reduce the brightness. Theoperator may cut highlights and may slide the third dot and move ithigher up so that the color correction is only receiving the highestbrightness.

The ranges can be edited so that the operator can edit, only extremelybright highlights on a scene with a lot of very bright flares, that mayonly occur in the 95-100% range of all brightnesses in the scene. Toachieve this the operator can select the “Highlights” mode, and thendial the third slider to 0.95 and leave the final slider at 1.0, so ascreating a range from 95%-100% for the final “highlights” section to actupon. Then, when editing Offset, Gamma, Gain, and other controls on thepanel, with Highlights buttons selected on 2468, those controls willonly edit the ‘top 5% bright colors’ in the scene.

The region slider 2470 allows the operator to target a specific sectionof the scene as opposed to the whole scene as a whole. For example, theoperator wants to edit only the brightness or darkness of scene. Theoperator may want to turn the gain down for the scene or just reduce thehighlights in the scenes. Therefore, the region slider 2470 acts like afilter on all of the color correction tools. It can affect all the colorwheels as well as that saturation and contrast controls that are belowit.

The operator can adjust the exposure slider 2472 to adjust the exposureof the selected region.

The operator can adjust the saturation slider 2474 to adjust thesaturation of the selected region.

The operator can adjust the contrast slider 2476 to adjust the contrastof the selected region.

The all projectors button 2478 applies the color correction to allprojectors. The all cameras button 2480 applies the correction to allcamera. The place button switches the user interface to the placementtab. The reset button 2484 removes all color corrections for theselected object. The max value 2486 sets a maximum value for the rangeof the exposure slider, should the operators want to push the outputvalues over the standard range.

FIG. 25 illustrates a first exemplary handles panel 2500. The handlespanel 2500 allows an operator to expose various attributes of virtualobjects on any node in the entire virtual scene. By exposing the variousattributes the operator is able to modify the attributes of the virtualobjects. For example, the virtual object can be a spaceship or a lightpost that will be illustrated in the virtual environment. The attributescan include a color, a position, an orientation, etc. This allows anoperator to select an object from a scene and dynamically adjust variousattributes for the object.

The handles panel 2500 allows the operator to store the attributes of avirtual object under a file name 2502. The operator can add or removevarious attribute types to the handles panel 2500 that can be adjusted.For example, the cube1 attribute 2506 and the transform1 attribute 2508are available to be added to the attribute editor under the handlespanel 2500.

The attributes can also be assigned to various groups (e.g.,handlegroup1). For example, motion capture stage origin(mocapStageOrigin) attribute 2510 and the transform1 attribute 2508, andvsRenderControl attribute 2512 can be assigned to the handlegroup 1.Certain attributes can be assigned to specifically to an operator (e.g.,my_handle) or a specific electronic device executing the handles panel2500. For example, the vsRendercontrol attribute 2512 can also beassigned to the my_handle group.

The selected attributes can be stored using a select filename 2514. Theattributes selected and listed in FIG. 25 are merely exemplary anddifferent configurations using different combinations of attributes canbe selected.

The float attribute 2516 can be used to control a positioning andformatting of content on a display. The operator can adjust the floatattribute 2516 by changing the float value 2518 or using a float slider2520.

The string attribute 2522 can assign a text value to a virtual object.The operator can adjust the string attribute 2522 using a text box 2524to enter a text string 2526.

A two-dimensional float vector (V2f) attribute 2528 (e.g., (w, h) or (x,y)) can control adjustment of a two dimensional (2D) positioning of avirtual object. The V2f attribute 2528 can include a V2f x-value 2530and a V2F y-value 2532. The V2f attribute can include a 2D aspect ratiolock 2534.

A three-dimensional float vector (V3f) attribute 2536 (e.g., x, y, andz) can control adjustment of a three dimensional (3D) positioning of avirtual object. The V3f attribute 2536 can include a V3f x-value 2538, aV3f y-value, a V3f z-value 2542. The V3f attribute can include a 3Daspect ratio lock 2544.

The Color3f attribute 2546 can control adjustment of a color of avirtual object. The Color3f attribute 2546 can include a Color3f redvalue 2548, a Color3f green value 2550, and a Color3f blue value 2552.The Color3f attribute can include a Color3f lock 2554. The Color3fattribute 2546 can include a color preview pane 2556.

The Color4f attribute 2558 can control adjustment of a color of avirtual object. The Color4f attribute 2558 can include a Color4f redvalue 2560, a Color4f green value 2562, a Color4f blue value 2564. TheColor3f attribute can include a Color4f lock 2566. The Color4f attribute2558 can include an alpha a-value 2568. The a-value 2568 can refer to anopacity. The Color4f attribute 2558 can include a color preview pane2570.

The enum attribute 2564 can be selected using the handles panel 2500. Anenum type can a distinct value type that declares a set of namedconstants. The operator can select option-1 2566, option-2 2568,option-3 2570, and option-4 2572 for the enum attribute 2564.

The Box2f attribute 2574 can be selected using the handles panel 2500.The Box2f attribute 2574 can include a Box2f x-value 2576, a Box2fy-value 2578, a Box2f width value 2580, and a Box2f height value 2582.

FIG. 26 illustrates a second exemplary handles panel 2600.

The handles panel 2600 allows the operator to store the attributes of avirtual object under a file name 2302. The operator can add or removevarious attribute types to the handles panel 2300 that can be adjusted.For example, the cube1 attribute 2306 and the transform1 attribute 2308are available to be added to the attribute editor on the handles panel2300.

The attributes can also be assigned to various groups (e.g.,handlegroup1). For example, motion capture stage origin(mocapStageOrigin) attribute 2310 and the transform1 attribute 2308, andvsRenderControl attribute 2312 can be assigned to the handlegroup 1.Certain attributes can be assigned to specifically to an operator (e.g.,my_handle) or a specific electronic device executing the handles panel2300. For example, the vsRendercontrol attribute 2312 can also beassigned to the my_handle group.

The selected attributes can be stored using a select filename 2314. Theattributes selected and listed in FIG. 26 are merely exemplary anddifferent configurations using different combinations of attributes canbe selected.

The operator can adjust a position 2602, a rotation 2604 and a scale2606 of the selected virtual object using the second exemplary handlespage 2600.

The operator can adjust the position 2602 of the virtual object usingthe longitude virtual wheel 2608, a distance coarse slider 2610, and adistance fine slider 2612. The operator can adjust the height of thevirtual object using the coarse height slider 2614 and the fine heightslider 2616.

The operator can select the axis aligned button 2618 to have themovement style of the virtual aligned with the axis. Alternatively, theoperator can select the origin button 2620 to have the movement style ofthe virtual object to be aligned with the origin. The operator can alsoadjust an x-value 2622, a y-value 2624, and a z-value 2626 for thevirtual object.

The operator can select a movement scale 2628 used for the longitudevirtual wheel 2608, a distance coarse slider 2610, and a distance fineslider 2612. In various embodiments, the movement scale can be variedfrom 0.1, 1, 10, 100, 1000 or an operator entered scale value 2630. Theoperator can select the done button 2632 to apply the changes to thevirtual object.

VII. Light Capture Device

In some instances, the mechanism may be a handheld pointing device. Thepointing device's orientation and position may be tracked using trackingcameras and/or sensors as it moves. The handheld pointing device mayinclude markers to facilitate said tracking. In some instances, thehandheld pointing device may have an activation button or trigger. Upona user activating the activation button or trigger, the contentproduction system may trigger a specific action in the immersive contentproduction system. The light capture device is more of apositional/remote control-type device, rather than a light measure/lightcapture device. The light capture device can capture light from theInfrared Mocap System Cameras, but the purpose of that is so that themotion capture system can accurately track the 6DOF (position androtation) of the light capture device in the real world, and so that theoperator can then use that data to perform remote-control/pointer-typeactions with the device. In doing so, the content production system candetermine a point or region of the immersive cave/wall at which thehandheld point device is being aimed or directed. The content productionsystem may, based on this determination, select the point or region ofthe immersive cave/wall. In other embodiments, the selection proceduremay include the immersive cave/wall displaying a reticle (e.g., acrosshair) corresponding to a region and/or point at which theaforementioned mechanisms are currently being directed. In doing so,selection of the region and/or point can be facilitated.

FIG. 27 illustrates a profile view of a light capture device 2700. Incertain embodiments, a user may use a light measuring device 2700 todetermine lighting and color values generated by the displays of theimmersive cave/wall. The light measuring device 2500 may provideinformation regarding the measured light and color values to the contentproduction system. The content production system may thereafter comparethe measured values with desired color and light values. The desiredvalues may have been measured at a physical location (i.e., on location)to be represented by the images displayed by the immersive cave/wall. Inother embodiments, the desired values may have been input by a user ofthe content production system. In some embodiments, the contentproduction system may iteratively adjust the lighting and color of theimmersive cave/wall in real time or substantially real-time (e.g., atinteractive frame rates) until (a) the measured light and color valuesand (b) the desired color and light values are within a threshold errorrange. The light capturing device 2700 can include a power source. Thepower source can be a battery. The battery can be a rechargeablebattery. The light capturing device 2700 can include a power connectionfor receiving power.

The light capturing device 2700 can include a body 2702 that can encasevarious electronics, sensors, and internal wiring. The body 2702 (e.g.,a housing) can include a portion shaped in the form of a grip 2704. Thegrip 2704 can allow the body 2702 to be held in the hand of a user. Thegrip 2704 allows for steady control of the light measuring device 2700and convenient one-handed access of various controls 2704 or a trigger2706. The controls 2704 can be multi-function buttons mapped to variousprogrammable functions for the immersive content production system. Forexample, the controls 2704 can be mapped to an on/off switch for avirtual green screen to turn the feature on and off. In variousembodiments, the controls can be used to resize a virtual green screen.In various embodiments, animations can be mapped to the controls 2704and can be triggered by activation. In various embodiments, the trigger2706 can be programmed for an action (e.g., an animation) to bedisplayed upon pressing the trigger 2706. In various embodiments, therecan be a jog wheel for making adjustments.

The light capturing device 2700 can include a plurality of light sensors2708 connected to various light sensor arms 2710 that can be affixed tothe body 2702 via a plurality of mounts 2712. The light sensors 2708 canbe active or passive sensors. The light sensors 2708 can be motiontracker markers. In various embodiments, the light sensors 2708 can beactive markers. The light sensors 2708 can be used by the immersivecontent production system to find the light capturing device 2700 in agiven space. The light sensor arms 2710 can extend at various angle andelevations from the body 2702 to enable the light sensors 2708 tocapture light at various points around the light capturing device 2700.In various embodiments, the light sensor arms 2710 can include variouslengths. In various embodiments, the light sensor arms 2710 can beflexible to enable the light sensor 2708 to be precisely positioned. Theconfiguration of the plurality of light sensors 2708 on the lightcapturing device 2700 can be known as a crown.

In various embodiments, multiple light capture devices 2700 can be usedon the same immersive content production set. In order for the motiontracking system to distinguish between the each of the multiple lightcapture devices 2700, a distinguishable or unique crown can be used foreach of the multiple light capture devices 2700. In various embodiments,the predetermined configuration can include a first light marker mountedto a first sensor arm that is mounted perpendicular to the handgrip onan opposite side of the housing. The predetermined configuration caninclude a plurality of additional markers arms with additional markersmounted thereon. The additional sensor arms can be mounted to theopposite side of the housing of the handgrip. The additional markers canbe mounted at distinct angles above and below the first sensor arm. Theadditional markers can be distributed in a distinct pattern.

The light capture device 2700 can include various universal mounts 2714.The universal mounts 2714 can enable various modules to be coupled withthe light capturing device 2700. In additional, the universal mounts2714 allow for different placement for mounting the plurality of lightsensors 2708.

The light capture device 2700 can include an electronic device mount2716 for removably coupling an electronic device 2718 (e.g., asmartphone). The electronic device 2718 can execute one or moreapplications for positioning the light measurement device 2700.

The light capture device 2700 can include one or more antenna 2720. Theantenna 2720 can wirelessly communicate data back to a control system(e.g., StageCraft) using any wireless protocol (e.g., Bluetooth, Wi-Fi,etc.). The data can be sent to a portion of the immersive content systemthat tracks the one or more cameras in the immersive content productionsystem (e.g., WVCAM). The data can include the position of the lightcapture device 2700. The antenna 2720 can be connected with atransmitter (not shown). In some embodiments, the antenna 2720 can beconnected with a transceiver. In various embodiments, the light capturedevice 2700 can include a controller 2722. The controller 2722 caninclude a circuit enclosed by the body (or housing). The circuit caninclude one or more processors, one or more memories, a transmitter, anda bus. In various embodiments, the controller 2722 can be an Arduinomicrocontroller based platform. The controller 2722 can receive inputsfrom the controls 2704 or the trigger 2706.

The light capture device 2700 has multiple use cases. In one use case,the light capture device 2700 can be used to draw the frustum of thetaking camera. The light capture device 2700 can simulate a position ofa camera and the MOCAP system can draw a position of the frustum forthat camera. This can be called “painting the frustum.”

Other use cases of the light capture device 2700 can include use forvirtual green screens. Actors can be performing on screen and the devicecan ensure that there is a green screen around certain features ofactors or costumes. The light capture device 2700 can be used to drawthe green screen behind the actors to help replace part of thebackground.

Other use cases for the light capture device 2700 can include placementof virtual stage lights. The light capture device can point in adirection where virtual stage lights are desired and the position andorientation can be sent from the light capture device 2700 to theimmersive content production system.

The light capture device 2700 cane be used as a virtual pointer tohighlight 3D points or 3D geometries (characters, set pieces) in ascene. The virtual pointer to can be triggered on certain objects in thescene. The orientation for the light capture device 2700 can bedetermined using the motion capture system.

The light capture device 2700 can be battery powered (e.g., one or morerechargeable batteries). The light capture device 2700 can include adirect current input as well.

FIG. 28 illustrates a top view of the light capture device 2700 thatillustrates a body 2702, the controls 2704, the plurality of lightsensors 2708 connected to various light sensor arms 2710 that can beaffixed to the body 2702 via the plurality of mounts 2712. FIG. 28 alsoillustrates the universal mount 2714, the electronic device mount 2716and the electronic device 2718. FIG. 28 also illustrates various antenna2720.

FIG. 29 illustrates a front view of the light capture device 2700 thatillustrates a body 2702, the controls 2704, the plurality of lightsensors 2708 connected to various light sensor arms 2710 that can beaffixed to the body 2702. FIG. 29 also illustrates the electronic devicemount 2716 and the display of the electronic device 2718. FIG. 29 alsoillustrates various antenna 2720 and the controller 2722.

FIG. 30 illustrates a profile view of the light capture device 2700 thatillustrates a body 2702, the controls 2704, the handle 2705, theplurality of light sensors 2708 connected to various light sensor arms2710 that can be affixed to the body 2702. FIG. 30 also illustrates theuniversal mount 2714, the electronic device mount 2716 and the displayof the electronic device 2718. FIG. 28 also illustrates various antenna2720 and the controller 2722.

VIII. Computing Device

Each of the embodiments disclosed herein can be implemented in aspecial-purpose computer system. FIG. 31 illustrates a computer system3100, in which various embodiments described herein can be implemented.The system 3100 can be used to implement any of the computer systemsdescribed above. As shown in the figure, computer system 3100 includes aprocessing unit 2904 that communicates with a number of peripheralsubsystems via a bus subsystem 3102. These peripheral subsystems caninclude a processing acceleration unit 3106, an I/O subsystem 3108, astorage subsystem 3118, and a communications subsystem 3124. Storagesubsystem 3118 includes tangible computer-readable storage media 3122and a system memory 3110.

Bus subsystem 3102 provides a mechanism for letting the variouscomponents and subsystems of computer system 3100 communicate with eachother as intended. Although bus subsystem 3102 is shown schematically asa single bus, alternative embodiments of the bus subsystem can utilizemultiple buses. Bus subsystem 3102 can be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Forexample, such architectures can include an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which can beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard.

Processing unit 3104, which can be implemented as one or more integratedcircuits (e.g., a conventional microprocessor or microcontroller),controls the operation of computer system 3100. One or more processorscan be included in processing unit 3104. These processors can includesingle core or multicore processors. In certain embodiments, processingunit 3104 can be implemented as one or more independent processing units3132 and/or sub processing unit 3134 with single or multicore processorsincluded in each processing unit. In other embodiments, processing unit3104 can also be implemented as a quad-core processing unit formed byintegrating two dual-core processors into a single chip.

In various embodiments, processing unit 3104 can execute a variety ofprograms in response to program code and can maintain multipleconcurrently executing programs or processes. At any given time, some orall of the program code to be executed can be resident in processor(s)3104 and/or in storage subsystem 3118. Through suitable programming,processor(s) 3104 can provide various functionalities described above.Computer system 3100 can additionally include a processing accelerationunit 3106, which can include a digital signal processor (DSP), aspecial-purpose processor, and/or the like. And, in some embodiments,the processing unit or another component of system 3100 can includeand/or operate a real-time gaming engine or other similar real-timerendering engine. Such an engine can render two-dimensional (2D) imagesfrom 3D data at interactive frame rates (e.g., 24, 48, 72, 96, or moreframes per second). In one aspect, the real-time gaming engine can loadthe virtual environment for display on the displays surrounding theperformance area. In some embodiments, the real-time gaming engine canload virtual assets into the virtual environment. The real-time gamingengine can then permit the virtual assets to interact or move accordingto simulated physics information stored by the real-time gaming engine.The real-time gaming engine can also update the virtual environmentbased on the movement and orientation of the taking camera(s).

I/O subsystem 3108 can include user interface input devices and userinterface output devices. User interface input devices can include akeyboard, pointing devices such as a mouse or trackball, a touchpad ortouch screen incorporated into a display, a scroll wheel, a click wheel,a dial, a button, a switch, a keypad, audio input devices with voicecommand recognition systems, microphones, and other types of inputdevices. User interface input devices can include, for example, motionsensing and/or gesture recognition devices such as the Microsoft Kinect®motion sensor that enables users to control and interact with an inputdevice, such as the Microsoft Xbox® 360 game controller, through anatural user interface using gestures and spoken commands. Userinterface input devices can also include eye gesture recognition devicessuch as the Google Glass® blink detector that detects eye activity(e.g., ‘blinking’ while taking pictures and/or making a menu selection)from users and transforms the eye gestures as input into an input device(e.g., Google Glass®). Additionally, user interface input devices caninclude voice recognition sensing devices that enable users to interactwith voice recognition systems (e.g., Siri® navigator), through voicecommands. In some embodiments, the user interface devices enable anoperator to provide input indicating the types of virtual assets and/oreffects to be integrated into the virtual environment displayed duringthe performance. The operator can also indicate the particularconfigurations or trigger movements of the performer and/or physicalobjects in the performance area that are to be used to begin the loadingand presentation of certain virtual assets. In some embodiments, theinput received from the operator can occur in real-time and/orconcurrently with a performance

The system 3100 can include one or more displays 3132. The displays 3132can be the displays 104 depicted in FIG. 1 . The displays 3132 can forman enclosed performance area. In some embodiments, the displays 3132 canbe formed from multiple light emitting diode (LED) panels. In someembodiments, the displays 3132 can be formed via multiple liquid crystaldisplay (LCD) panels or thin-film transistor liquid-crystal display (TFTLCD) panels.

The system 3100 can include one or more cameras 3134. The one or morecameras can be digital cameras. Digital cinematography captures motionpictures digitally in a process analogous to digital photography.Professional cameras can include the Sony CineAlta(F) Series, BlackmagicCinema Camera, RED ONE, Arriflex D-20, D-21 and Alexa, PanavisionsGenesis, Silicon Imaging SI-2K, Thomson Viper, Vision Research Phantom,IMAX 3D camera based on two Vision Research Phantom cores, Weisscam HS-1and HS-2, GS Vitec noX, and the Fusion Camera System. Digitalcinematography cameras can capture images using complementarymetal-oxide semiconductor (CMOS) or charge coupled device (CCD) sensors,usually in one of two arrangements. Single chip cameras that aredesigned specifically for the digital cinematography market often use asingle sensor (much like digital photo cameras), with dimensions similarin size to a 16 or 35 mm film frame or even (as with the Vision 65) a 65mm film frame. An image can be projected onto a single large sensorexactly the same way it can be projected onto a film frame, so cameraswith this design can be made with positive lock (PL), Panavision (PV)and similar mounts, in order to use the wide range of existing high-endcinematography lenses available. Their large sensors also let thesecameras achieve the same shallow depth of field as 35 or 65 mm motionpicture film cameras, which many cinematographers consider an essentialvisual tool.

Unlike other video formats, which are specified in terms of verticalresolution (for example, 1080p, which is 1920×1080 pixels), digitalcinema formats are usually specified in terms of horizontal resolution.As a shorthand, these resolutions are often given in “nK” notation,where n is the multiplier of 3124 such that the horizontal resolution ofa corresponding full-aperture, digitized film frame is exactly 1024npixels.

For instance, a 2K image is 2048 pixels wide, and a 4K image is 4096pixels wide. Vertical resolutions vary with aspect ratios though; so a2K image with an HDTV (16:9) aspect ratio is 2048×1152 pixels, while a2K image with a standard definition television (SDTV) or Academy ratio(4:3) is 2048×1536 pixels, and one with a Panavision ratio (2.39:1)would be 2048×856 pixels, and so on. Due to the “nK” notation notcorresponding to specific horizontal resolutions per format a 2K imagelacking, for example, the typical 35 mm film soundtrack space, is only1828 pixels wide, with vertical resolutions rescaling accordingly.

All formats designed for digital cinematography are progressive scan,and capture usually occurs at the same 24 frame per second rateestablished as the standard for 35 mm film. Some films have a High FrameRate of 48 fps, although most traditional theaters use 24 fps. The DCIstandard for cinema usually relies on a 1.89:1 aspect ratio, thusdefining the maximum container size for 4K as 4096×2160 pixels and for2K as 2048×1080 pixels.

Broadly, several workflow paradigms can be used for data acquisition andstorage in digital cinematography. With video-tape-based workflow, videois recorded to tape on set. This video is then ingested into a computerrunning non-linear editing software, using a deck. Upon ingestion, adigital video stream from tape is converted to computer files. Thesefiles can be edited directly or converted to an intermediate format forediting. Then video is output in its final format, possibly to a filmrecorder for theatrical exhibition, or back to video tape for broadcastuse. Original video tapes are kept as an archival medium. The filesgenerated by the non-linear editing application contain the informationnecessary to retrieve footage from the proper tapes, should the footagestored on the computer's hard disk be lost. With increasing convenienceof file-based workflows, the tape-based workflows have become marginalin recent years.

Digital cinematography can use tapeless or file-based workflows. Thistrend has accelerated with increased capacity and reduced cost ofnon-linear storage solutions such as hard disk drives, optical discs,and solid-state memory. With tapeless workflows digital video isrecorded as digital files onto random-access media like optical discs,hard disk drives or flash memory-based digital magazines. These filescan be easily copied to another storage device, typically to a largeRAID (array of computer disks) connected to an editing system. Once datais copied from the on-set media to the storage array, they are erasedand returned to the set for more shooting.

Such RAID arrays, both of managed (for example, storage area networks(SANs) and networked attached storage (NASs) and unmanaged (for example,just a bunch of disks (JBoDs) on a single computer workstation), arenecessary due to the throughput required for real-time (320 Megabits persecond for 2K @ 24 frames per second) or near-real-time playback inpost-production, compared to throughput available from a single, yetfast, hard disk drive. Such requirements are often termed as on-line orcloud storage. Post-production not requiring real-time playbackperformances (typically for lettering, subtitling, versioning and othersimilar visual effects) can be migrated to slightly slower RAID stores.

Short-term archiving, if ever, is accomplished by moving the digitalfiles into slower redundant array of independent disks (RAID) arrays(still of either managed or unmanaged type, but with lowerperformances), where playback capability is poor to non-existent (unlessvia proxy images), but minimal editing and metadata harvesting stillfeasible. Such intermediate requirements easily fall into the mid-linestorage category.

Long-term archiving is accomplished by backing up the digital files fromthe RAID, using standard practices and equipment for data backup fromthe information technology industry, often to data tapes (like lineartape open (LTOs)).

The system can include one or more spherical cameras. A spherical cameracan be called an omnidirectional camera, also known as 360-degreecamera, is a camera having a field of view that covers approximately theentire sphere or at least a full circle in the horizontal plane.360-degree videos, also known as immersive videos, or spherical videos,are video recordings where a view in every direction is recorded at thesame time, shot using an omnidirectional camera or a collection ofcameras. During playback on normal flat display the viewer has controlof the viewing direction like a panorama. It can also be played on adisplays or projectors arranged in a sphere or some part of a sphere.

360-degree video is typically recorded using either a special rig ofmultiple cameras, or using a dedicated camera that contains multiplecamera lenses embedded into the device, and filming overlapping anglessimultaneously. Through a method known as video stitching, this separatefootage is merged into one spherical video piece, and the color andcontrast of each shot is calibrated to be consistent with the others.This process is done either by the camera itself, or using specializedsoftware that can analyze common visuals and audio to synchronize andlink the different camera feeds together. Generally, the only area thatcannot be viewed is the view toward the camera support.

360-degree video is typically formatted in an equi-rectangularprojection and is either monoscopic, with one image directed to botheyes, or stereoscopic, viewed as two distinct images directedindividually to each eye for a 3D effect. Due to this projection andstitching, equi-rectangular video exhibits a lower quality in the middleof the image than at the top and bottom.

Specialized omnidirectional cameras and rigs have been developed for thepurpose of filming 360-degree video, including rigs such as GoPro's Omniand Odyssey (which consist of multiple action cameras installed within aframe), and contained cameras like the HumanEyes Vuze and Nokia OZO,There have also been handheld dual-lens cameras such as the Ricoh ThetaS, Samsung Gear 360, Garmin VIRB 360, and the Kogeto Dot 360—a panoramiccamera lens accessory developed for the iPhone 4, 4S, and Samsung GalaxyNexus.

User interface input devices can also include, without limitation, threedimensional (3-D) mice, joysticks or pointing sticks, gamepads andgraphic tablets, and audio/visual devices such as speakers, digitalcameras, digital camcorders, portable media players, webcams, imagescanners, fingerprint scanners, barcode reader 3-D scanners, 3-Dprinters, laser rangefinders, and eye gaze monitoring devices.Additionally, user interface input devices can include, for example,medical imaging input devices such as computed tomography, magneticresonance imaging, position emission tomography, medical ultrasonographydevices. User interface input devices can also include, for example,audio input devices such as MIDI keyboards, digital musical instrumentsand the like.

User interface output devices can include a display subsystem, indicatorlights, or non-visual displays such as audio output devices, etc. Thedisplay subsystem can be a cathode ray tube (CRT), a flat-panel device,such as that using a liquid crystal display (LCD) or plasma display, aprojection device, a touch screen, and the like. In general, use of theterm “output device” is intended to include all possible types ofdevices and mechanisms for outputting information from computer system3100 to a user or other computer. For example, user interface outputdevices can include, without limitation, a variety of display devicesthat visually convey text, graphics and audio/video information such asmonitors, printers, speakers, headphones, automotive navigation systems,plotters, voice output devices, and modems.

Computer system 3100 can comprise a storage subsystem 3118 thatcomprises software elements, shown as being currently located within asystem memory 3110. System memory 3110 can store program instructionsthat are loadable and executable on processing unit 3104, as well asdata generated during the execution of these programs.

Depending on the configuration and type of computer system 3100, systemmemory 3110 can be volatile (such as random access memory (RAM)) and/ornon-volatile (such as read-only memory (ROM), flash memory, etc.) TheRAM typically contains data and/or program modules that are immediatelyaccessible to and/or presently being operated and executed by processingunit 3104. In some implementations, system memory 3110 can includemultiple different types of memory, such as static random access memory(SRAM) or dynamic random access memory (DRAM). In some implementations,a basic input/output system (BIOS), containing the basic routines thathelp to transfer information between elements within computer system3100, such as during start-up, can typically be stored in the ROM. Byway of example, and not limitation, system memory 3110 also illustratesapplication programs 3112, which can include client applications, webbrowsers, mid-tier applications, relational database management systems(RDBMS), etc., program data 3114, and an operating system 3116. By wayof example, operating system 3116 can include various versions ofMicrosoft Windows®, Apple Macintosh®, and/or Linux operating systems, avariety of commercially-available UNIX® or UNIX-like operating systems(including without limitation the variety of GNU/Linux operatingsystems, the Google Chrome® OS, and the like) and/or mobile operatingsystems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, andPalm® OS operating systems.

Storage subsystem 3118 can also provide a tangible computer-readablestorage medium for storing the basic programming and data constructsthat provide the functionality of some embodiments. Software (programs,code modules, instructions) that when executed by a processor providethe functionality described above can be stored in storage subsystem3118. These software modules or instructions can be executed byprocessing unit 3104. Storage subsystem 3118 can also provide arepository for storing data used in accordance with the presentinvention.

Storage subsystem 3100 can also include a computer-readable storagemedia reader 3120 that can further be connected to computer-readablestorage media 3122. Together and, optionally, in combination with systemmemory 3110, computer-readable storage media 3122 can comprehensivelyrepresent remote, local, fixed, and/or removable storage devices plusstorage media for temporarily and/or more permanently containing,storing, transmitting, and retrieving computer-readable information.

Computer-readable storage media 3122 containing code, or portions ofcode, can also include any appropriate media known or used in the art,including storage media and communication media, such as but not limitedto, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information. This can include tangible computer-readable storagemedia such as RAM, ROM, electronically erasable programmable ROM(EEPROM), flash memory or other memory technology, CD-ROM, digitalversatile disk (DVD), or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or other tangible computer readable media. This can also includenontangible computer-readable media, such as data signals, datatransmissions, or any other medium which can be used to transmit thedesired information and which can be accessed by computing system 3100.

By way of example, computer-readable storage media 3122 can include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 3122 can include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 3122 can also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media can providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 3100.

Communications subsystem 3124 provides an interface to other computersystems and networks. Communications subsystem 3124 serves as aninterface for receiving data from and transmitting data to other systemsfrom computer system 3100. For example, communications subsystem 3124can enable computer system 3100 to connect to one or more devices viathe Internet. In some embodiments communications subsystem 3124 caninclude radio frequency (RF) transceiver components for accessingwireless voice and/or data networks (e.g., using cellular telephonetechnology, advanced data network technology, such as 3G, 4G or EDGE(enhanced data rates for global evolution), Wi-Fi (IEEE 802.11 familystandards, or other mobile communication technologies, or anycombination thereof), global positioning system (GPS) receivercomponents, and/or other components. In some embodiments communicationssubsystem 3124 can provide wired network connectivity (e.g., Ethernet)in addition to or instead of a wireless interface.

In some embodiments, communications subsystem 3124 can also receiveinput communication in the form of structured and/or unstructured datafeeds 3126, event streams 3128, event updates 3130, and the like onbehalf of one or more users who can use computer system 3100.

By way of example, communications subsystem 3124 can be configured toreceive data feeds 3126 in real-time from users of social networksand/or other communication services such as Twitter® feeds, Facebook®updates, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources.

Additionally, communications subsystem 3124 can also be configured toreceive data in the form of continuous data streams, which can includeevent streams 3128 of real-time events and/or event updates 3130, thatcan be continuous or unbounded in nature with no explicit end. Examplesof applications that generate continuous data can include, for example,sensor data applications, financial tickers, network performancemeasuring tools (e.g. network monitoring and traffic managementapplications), clickstream analysis tools, automobile trafficmonitoring, and the like.

Communications subsystem 3124 can also be configured to output thestructured and/or unstructured data feeds 3126, event streams 3128,event updates 3130, and the like to one or more databases that can be incommunication with one or more streaming data source computers coupledto computer system 3100.

Computer system 3100 can be one of various types, including a handheldportable device (e.g., an iPhone® cellular phone, an iPad® computingtablet, a PDA), a wearable device (e.g., a Google Glass® head mounteddisplay), a PC, a workstation, a mainframe, a kiosk, a server rack, orany other data processing system.

Due to the ever-changing nature of computers and networks, thedescription of computer system 3100 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software (includingapplets), or a combination. Further, connection to other computingdevices, such as network input/output devices, can be employed. Based onthe disclosure and teachings provided herein, a person of ordinary skillin the art will appreciate other ways and/or methods to implement thevarious embodiments.

In the foregoing description, for the purposes of explanation, numerousspecific details were set forth in order to provide a thoroughunderstanding of various embodiments of the present invention. It willbe apparent, however, to one skilled in the art that embodiments of thepresent invention can be practiced without some of these specificdetails. In other instances, well-known structures and devices are shownin block diagram form.

The foregoing description provides exemplary embodiments only, and isnot intended to limit the scope, applicability, or configuration of thedisclosure. Rather, the foregoing description of the exemplaryembodiments will provide those skilled in the art with an enablingdescription for implementing an exemplary embodiment. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the spirit and scope ofthe invention as set forth in the appended claims.

Specific details are given in the foregoing description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments canbe practiced without these specific details. For example, circuits,systems, networks, processes, and other components may have been shownas components in block diagram form in order not to obscure theembodiments in unnecessary detail. In other instances, well-knowncircuits, processes, algorithms, structures, and techniques may havebeen shown without unnecessary detail in order to avoid obscuring theembodiments.

Also, it is noted that individual embodiments may have been described asa process which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay have described the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations can be re-arranged. A process is terminatedwhen its operations are completed, but could have additional steps notincluded in a figure. A process can correspond to a method, a function,a procedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

The term “computer-readable medium” includes, but is not limited toportable or fixed storage devices, optical storage devices, wirelesschannels and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A code segment ormachine-executable instructions can represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment can be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc., can be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

Furthermore, embodiments can be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks can be stored in a machine readable medium. A processor(s) canperform the necessary tasks.

In the foregoing specification, aspects of the invention are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the invention is not limited thereto. Variousfeatures and aspects of the above-described invention can be usedindividually or jointly. Further, embodiments can be utilized in anynumber of environments and applications beyond those described hereinwithout departing from the broader spirit and scope of thespecification. The specification and drawings are, accordingly, to beregarded as illustrative rather than restrictive.

The foregoing description provides exemplary embodiments only, and isnot intended to limit the scope, applicability, or configuration of thedisclosure. Rather, the foregoing description of the exemplaryembodiments will provide those skilled in the art with an enablingdescription for implementing an exemplary embodiment. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the spirit and scope ofthe invention as set forth in the appended claims.

Specific details are given in the foregoing description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments canbe practiced without these specific details. For example, circuits,systems, networks, processes, and other components may have been shownas components in block diagram form in order not to obscure theembodiments in unnecessary detail. In other instances, well-knowncircuits, processes, algorithms, structures, and techniques may havebeen shown without unnecessary detail in order to avoid obscuring theembodiments.

Also, it is noted that individual embodiments may have been described asa process which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay have described the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations can be re-arranged. A process is terminatedwhen its operations are completed, but could have additional steps notincluded in a figure. A process can correspond to a method, a function,a procedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

The term “computer-readable medium” includes, but is not limited toportable or fixed storage devices, optical storage devices, wirelesschannels and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A code segment ormachine-executable instructions can represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment can be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc., can be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

Furthermore, embodiments can be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks can be stored in a machine-readable medium. A processor(s) canperform the necessary tasks.

In the foregoing specification, aspects of the invention are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the invention is not limited thereto. Variousfeatures and aspects of the above-described invention can be usedindividually or jointly. Further, embodiments can be utilized in anynumber of environments and applications beyond those described hereinwithout departing from the broader spirit and scope of thespecification. The specification and drawings are, accordingly, to beregarded as illustrative rather than restrictive.

What is claimed is:
 1. A computer-implemented method of generatingcontent, the method comprising: generating and displaying images of avirtual environment on one or more light-emitting diode (LED) displaysat least partially surrounding a performance area; capturing, by ataking camera, a plurality of images of a performer or a physical objectin the performance area, wherein the performer or the physical object isin a foreground of the plurality of captured images and at least aportion of the performer or the physical object is displayed in imagesof the virtual environment in a background of the plurality of capturedimages; receiving one or more inputs identifying a color mismatchbetween the performer or the physical object in the foreground and theportion of the performer or the physical object displayed in the imagesof the virtual environment in the background; generating a patch for theimages of the virtual environment to correct the color mismatch;inserting the patch into the images of the virtual environment displayedon the one or more light-emitting diode (LED) displays; and generatingcontent based on the plurality of captured images.
 2. Thecomputer-implemented method of claim 1, wherein the patch is displayedas a two-dimensional image that is flat with respect to the one or moreLED displays.
 3. The computer-implemented method of claim 1, furthercomprising adjusting an intensity of the patch displayed on the one ormore LED displays.
 4. The computer-implemented method of claim 1,further comprising adjusting a position of the patch displayed on theone or more LED displays.
 5. The computer-implemented method of claim 1,further comprising adjusting an orientation of the patch displayed onthe one or more LED displays.
 6. The computer-implemented method ofclaim 1, further comprising adjusting an exposure of the patch displayedon the one or more LED displays.
 7. The computer-implemented method ofclaim 1, further comprising adjusting a size of the patch displayed onthe one or more LED displays.
 8. A non-transitory computer-readablemedium storing a set of instructions, the set of instructionscomprising: one or more instructions that, when executed by one or moreprocessors of a computing device, cause the computing device to:generate and display images of a virtual environment on one or morelight-emitting diode (LED) displays at least partially surrounding aperformance area; capture, by a taking camera, a plurality of images ofa performer or a physical object in the performance area, wherein theperformer or the physical object is in a foreground of the plurality ofcaptured images and at least a portion of the performer or the physicalobject is displayed in images of the virtual environment in a backgroundof the plurality of captured images; receive one or more inputs thatidentify a color mismatch between the performer or the physical objectin the foreground and the portion of the performer or the physicalobject displayed in the images of the virtual environment in thebackground; generate a patch for the images of the virtual environmentto correct the color mismatch; insert the patch into the images of thevirtual environment displayed on the one or more LED displays; andgenerate content based on the plurality of captured images.
 9. Thenon-transitory computer-readable medium of claim 8, wherein the patch isdisplayed as a two-dimensional image that is flat with respect to theone or more LED displays.
 10. The non-transitory computer-readablemedium of claim 8, wherein the one or more instructions further causethe computing device to adjust an intensity of the patch displayed onthe one or more LED displays.
 11. The non-transitory computer-readablemedium of claim 8, wherein the one or more instructions further causethe computing device to adjust a position of the patch displayed on theone or more LED displays.
 12. The non-transitory computer-readablemedium of claim 8, wherein the one or more instructions further causethe computing device to adjust an orientation of the patch displayed onthe one or more LED displays.
 13. The non-transitory computer-readablemedium of claim 8, wherein the one or more instructions further causethe computing device to adjust an exposure of the patch displayed on theone or more LED displays.
 14. The non-transitory computer-readablemedium of claim 8, wherein the one or more instructions further causethe computing device to adjust a size of the patch displayed on the oneor more LED displays.
 15. An immersive content generation system,comprising: one or more memories; and one or more processors,communicatively coupled to the one or more memories, configured to:generate and display images of a virtual environment on one or morelight-emitting diode (LED) displays at least partially surrounding aperformance area; capture, by a taking camera, a plurality of images ofa performer or a physical object in the performance area, wherein theperformer or the physical object is in a foreground of the plurality ofcaptured images and at least a portion of the performer or the physicalobject is displayed in images of the virtual environment in a backgroundof the plurality of captured images; receive one or more inputs thatidentify a color mismatch between the performer or the physical objectin the foreground and the portion of the performer or the physicalobject displayed in the images of the virtual environment in thebackground; generate a patch for the images of the virtual environmentto correct the color mismatch; insert the patch into the images of thevirtual environment displayed on the one or more LED displays; andgenerate content based on the plurality of captured images.
 16. Theimmersive content generation system of claim 15, wherein the patch isdisplayed as a two-dimensional image that is flat with respect to theone or more LED displays.
 17. The immersive content generation system ofclaim 15, wherein the one or more processors are further configured toadjust an intensity of the patch displayed on the one or more LEDdisplays.
 18. The immersive content generation system of claim 15,wherein the one or more processors are further configured to adjust aposition of the patch displayed on the one or more LED displays.
 19. Theimmersive content generation system of claim 15, wherein the one or moreprocessors are further configured to adjust an orientation of the patchdisplayed on the one or more LED displays.
 20. The immersive contentgeneration system of claim 15, wherein the one or more processors arefurther configured to adjust an exposure of the patch displayed on theone or more LED displays.